Multimode fusion and structured knowledge driven complex file understanding method and system

Abstract

The invention discloses a multi-mode fusion and structured knowledge driven complex file understanding method and system. The method comprises the steps of extracting multi-mode features for a complex file; mapping multi-modal features in a high-dimensional semantic space, and constructing structured semantic data; using a domain knowledge base to retrieve background information of the structured semantic data; and generating a target file based on the structured semantic data and the corresponding background information. Therefore, multi-modal data are effectively extracted through multi-modal feature extraction and a cross-modal attention alignment mechanism, semantic alignment and adaptive weight distribution are further realized in a high-dimensional semantic space, multi-modal information consistency is ensured, semantic faults and information conflicts are eliminated, and structured knowledge driving is assisted, so that the multi-modal data are effectively extracted. According to the method, the analysis capability of professional terms, implicit logic and context associated content is greatly enhanced, the output quality is high, accurate output can be achieved for different application scenes, and the application range is wide.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a complex document understanding method and system driven by multi-modal fusion and structured knowledge. Background Art

[0002] Currently, the field of big data requires the collection of training data, which is obtained by processing massive amounts of complex files. This large-scale collection of training data exhibits multimodal and highly complex characteristics. For example, even complex files collected for the same needs may appear in different forms, such as tables, charts, images, and formulas, depending on the file format requirements, editors, and release dates. Furthermore, the same complex file may contain data in the form of technical parameter text, performance indicator charts, and explanatory formulas. These multimodal characteristics are difficult to standardize, and existing automated collection solutions struggle to establish effective associations between information in different modalities, resulting in high rates of misjudgment and omission of key information.

[0003] Moreover, since different modal data are expressed in different forms, such as text as word sequences, images as pixel matrices, and charts as row and column structures, existing automated collection solutions find it difficult to establish accurate cross-modal semantic mapping relationships. The converted data have semantic gaps, which makes information fusion difficult in subsequent processing and even causes information conflicts between different modal data.

[0004] In addition, existing automated collection solutions mostly use text recognition and image recognition, lack the ability to perform associative retrieval of domain knowledge in complex documents, and fail to effectively integrate multimodal information and domain knowledge. Therefore, they perform poorly when processing professional terms or implicit logic. The generated summaries are often superficial descriptions, affecting the quality of the output results. They cannot output in a targeted manner according to training needs and application scenarios, limiting the diversification of application scenarios. Summary of the Invention

[0005] A first aspect of an embodiment of the present invention discloses a complex document understanding method driven by multimodal fusion and structured knowledge, specifically comprising:

[0006] Extract multimodal features for complex files;

[0007] Mapping the multimodal features in a high-dimensional semantic space to construct structured semantic data;

[0008] Retrieving background information of the structured semantic data using a domain knowledge base;

[0009] A target file is generated based on the structured semantic data and corresponding background information.

[0010] As an optional implementation, the extracting multimodal features for complex files includes:

[0011] Perform word segmentation, part-of-speech tagging, and stop word removal on text data to generate standardized characters;

[0012] Identify the row and column structure, table headers, and data cells of the table data, and analyze the semantic associations between the data cells to establish a reference relationship between the table data and surrounding descriptions;

[0013] Obtain multi-level visual features including coordinate axes, legends, and data points for chart data, and generate natural language descriptions;

[0014] A segmentation network is used to locate the operators and scalars of formula data, and the corresponding syntax tree representation is generated based on the formal grammar.

[0015] As an optional implementation, mapping the multimodal features in a high-dimensional semantic space to construct structured semantic data includes:

[0016] In the high-dimensional semantic space, a multi-head attention mechanism is used to calculate the correlation between different modal features to generate an inter-modal correlation matrix;

[0017] Based on the association matrix, the contribution weight of each modal feature in the fusion process is dynamically adjusted through learnable weight parameters.

[0018] As an optional implementation, the method further includes:

[0019] The calculation formula of the multi-head attention mechanism is as follows:

[0020]

[0021] Q=XW Q , K=XW K , V=XW V ;

[0022] Among them, X is the multimodal splicing feature;

[0023] dk represents the dimension scaling factor, where the square root of the feature dimension is taken;

[0024] Matrix addition calculation is used to implement attention bias on the correlation matrix.

[0025] As an optional implementation, the retrieving background information of the structured semantic data using a domain knowledge base includes:

[0026] Identifying core topics and key entities in the structured semantic data, wherein the key entities include at least professional terms, organization names, and specific concepts;

[0027] Extracting semantic associations between the core topics and the key entities through a relationship extraction algorithm, including explicit annotation relationships and implicit relationships based on contextual reasoning;

[0028] Constructing a query instruction based on the core topic and the key entity;

[0029] Based on the query instruction, a targeted search is performed in the domain knowledge base to obtain background information including knowledge graph fragments, background knowledge, industry standards, professional term definitions or domain-specific rules.

[0030] As an optional implementation, the semantic association between the core topic and the key entity is extracted by a relationship extraction algorithm, including explicit annotation relationships and implicit relationships based on contextual reasoning, including:

[0031] A graph neural network is used to transmit messages and update node representations on the knowledge graph composed of the core topics and key entities, thereby realizing dynamic knowledge reasoning based on the graph structure.

[0032] Wherein, the dynamic knowledge reasoning integrates contextual semantic information.

[0033] As an optional implementation manner, generating a target file based on the structured semantic data and corresponding background information includes:

[0034] Building a sequence-to-sequence model based on the Transformer encoder;

[0035] Using the explicit or implicit relationships obtained by the graph neural network reasoning as joint input to obtain high-weight information;

[0036] Combining the structured semantic data and enhanced feature representation, a target file that meets domain logic and specification requirements is generated.

[0037] As an optional implementation, the method further includes:

[0038] Preset a number of scene portraits, each of which has a corresponding generation strategy;

[0039] The generation strategy includes professional depth weight, information strength control rules, language style template and user interaction protocol, and the generation strategy of different scene portraits has differentiated settings;

[0040] According to the selected scenario portrait, the target file is optimized and output based on its corresponding generation strategy.

[0041] As an optional implementation, the method further includes:

[0042] Obtaining missing data marked for the target file;

[0043] The weight of the original data corresponding to the missing data is adjusted in the complex file, and is re-placed into the high-dimensional semantic space for mapping.

[0044] A second aspect of an embodiment of the present invention discloses a complex document understanding system driven by multi-modal fusion and structured knowledge, including:

[0045] Feature extraction module, used to extract multimodal features for complex files;

[0046] A multimodal fusion module is used to map the multimodal features in a high-dimensional semantic space and construct structured semantic data through a multi-head attention mechanism;

[0047] A knowledge fusion module, configured to retrieve background information of the structured semantic data using a domain knowledge base;

[0048] The generation module is used to generate a target file based on the structured semantic data and the corresponding background information, and output a visual map to the user terminal according to the scene portrait and its generation strategy.

[0049] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0050] Through multimodal feature extraction and cross-modal attention alignment mechanism, multimodal data can be effectively extracted, and then semantic alignment and adaptive weight distribution can be achieved in high-dimensional semantic space to ensure the consistency of multimodal information, eliminate semantic faults and information conflicts, and supplemented by structured knowledge drive. The ability to parse professional terminology, implicit logic and context-related content is greatly enhanced, with high output quality, accurate output for current application scenarios, and a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 This is a workflow diagram of a complex document understanding method driven by multi-modal fusion and structured knowledge disclosed in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the system structure of a complex document understanding system driven by multi-modal fusion and structured knowledge disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first," "second," "third," "fourth," etc. in the description and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having," as well as any variations thereof, in the embodiments of the present invention, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products, or apparatuses.

[0056] Example 1

[0057] See also Figure 1 ,like Figure 1 As shown, a complex document understanding method driven by multi-modal fusion and structured knowledge disclosed in an embodiment of the present invention includes the following steps:

[0058] 101. Extract multimodal features for complex files.

[0059] In this embodiment, taking the research report of the real estate industry as an example, complex files refer to report files issued by many real estate developers, contractors, research institutes and other units and institutions. The collected report files may be in different formats such as Word files, PDF files, and image files.

[0060] Among them, there are significant differences in the writing format and content arrangement of the report documents issued by different units and institutions.

[0061] In addition, even report files published by the same unit or organization may contain multimodal information such as tables, charts, pictures, formulas, etc. in addition to text information.

[0062] As an optional implementation, word segmentation, part-of-speech tagging, and stop word removal are performed on the text data to generate standardized characters;

[0063] Identify the row and column structure, table headers, and data cells of table data, analyze the semantic associations between data cells, and establish reference relationships between table data and surrounding descriptions;

[0064] Obtain multi-level visual features including coordinate axes, legends, and data points for chart data, and generate natural language descriptions;

[0065] A segmentation network is used to locate the operators and scalars of formula data, and the corresponding syntax tree representation is generated based on the formal grammar.

[0066] For example, if the report involves the drug dosage calculation formula "D=W*C", where D is the drug dosage, W is the patient's weight, and C is the dosage coefficient, the formula is converted into LaTeX code and then parsed into a structured mathematical expression.

[0067] Here, for complex files, a targeted solution is adopted to extract multimodal features, so that valid information in different forms is converted into string information in the same format, which is convenient for subsequent analysis and processing.

[0068] 102. Map multimodal features in high-dimensional semantic space to construct structured semantic data.

[0069] In this embodiment, by eliminating format barriers of various types of valid information in complex files, structured vector data with standardized specifications is constructed.

[0070] As an optional implementation, a multi-head attention mechanism is used to calculate the correlation between different modal features in a high-dimensional semantic space to generate an inter-modal correlation matrix;

[0071] Based on the association matrix, the contribution weight of each modal feature in the fusion process is dynamically adjusted through learnable weight parameters.

[0072] Specifically, the calculation formula of the multi-head attention mechanism is as follows:

[0073]

[0074] Q=XW Q , K=XW K , V=XW V ;

[0075] Among them, X is the multimodal splicing feature;

[0076] dk represents the dimension scaling factor, where the square root of the feature dimension is taken;

[0077] Matrix addition is used to implement attention bias on the correlation matrix.

[0078] Here, in the high-dimensional semantic space, the multi-head attention mechanism is used to adjust the weight of the key content in the complex document.

[0079] 103. Use domain knowledge base to retrieve background information of structured semantic data.

[0080] In this embodiment, the background information can clarify the specific fields involved in the structured semantic data, so as to avoid invalid searches in irrelevant fields due to similar keywords, thereby wasting computing power.

[0081] As an optional implementation, semantic associations between core topics and key entities are extracted through a relationship extraction algorithm, including explicit annotation relationships and implicit relationships based on contextual reasoning;

[0082] Build query instructions based on core topics and key entities;

[0083] Based on the query instructions, targeted retrieval is performed in the domain knowledge base to obtain background information including knowledge graph fragments, background knowledge, industry standards, professional term definitions or domain-specific rules.

[0084] Here, taking "ant colony" as an example, it is used in the field of biology to represent a biological settlement formed by many ants, while in the field of algorithms there is the "ant colony algorithm", which is used to represent a probabilistic algorithm for seeking path optimization. Therefore, on the basis of identifying "ant colony" as the core theme, other key entities should also be extracted and queried to clarify the explicit or implicit relationship between them, and then clarify their specific fields to achieve accurate positioning of structured semantic data.

[0085] Taking a medical research report as an example, if there is a sentence in the report that says "blood sugar control in diabetic patients is closely related to the use of [insulin]", then "diabetes" and "insulin" are identified as key entities.

[0086] Assuming that the medical research report mainly discusses the treatment and management of diabetes, then according to the document topic, relevant knowledge graphs and background information about the pathogenesis of diabetes, different treatments and their effects, and common complications can be retrieved.

[0087] When a text mentions "the patient's insulin resistance is severe," the system leverages knowledge from the domain knowledge base to understand the meaning of the term "insulin resistance." Furthermore, based on the relationship between insulin resistance and diabetes treatment outcomes and complications in the knowledge graph, the system can enhance its analysis of relevant content in the document. For example, the knowledge base might indicate that insulin resistance can reduce a patient's sensitivity to hypoglycemic medications, helping the system better understand subsequent content in the report regarding adjustments to medication dosages.

[0088] As an optional implementation, a graph neural network is used to transmit messages and update node representations on the knowledge graph composed of core topics and key entities, realizing dynamic knowledge reasoning based on the graph structure.

[0089] Among them, dynamic knowledge reasoning integrates contextual semantic information.

[0090] Here, integrating contextual semantic information can effectively identify implicit logic and improve the quality of output results.

[0091] As another optional implementation, a sequence-to-sequence model is established based on the Transformer encoder;

[0092] Use explicit or implicit relationships obtained by graph neural network reasoning as joint input to obtain high-weight information;

[0093] Combine structured semantic data and enhanced feature representation to generate target files that meet domain logic and specification requirements.

[0094] Here, we further improve the logic of the output results based on the association relationship and increase the information density contained in the text, rather than outputting long and straightforward sentences.

[0095] 104. Generate a target file based on the structured semantic data and corresponding background information.

[0096] In this embodiment, after effectively extracting multimodal features and clarifying their background information, the target file corresponding to the requirements can be output.

[0097] As an optional implementation, several scene portraits are preset, and each scene portrait has a corresponding generation strategy;

[0098] The generation strategy includes professional depth weight, information intensity control rules, language style templates, and user interaction protocols. The generation strategies for different scenario portraits have differentiated settings.

[0099] According to the selected scenario portrait, the target file is optimized and output based on its corresponding generation strategy.

[0100] Here, after ensuring the validity of the target file's information and the accuracy of its background, you can also set up a scene portrait for users to flexibly select and personalize the output of the target file.

[0101] For example, for the same astronomy paper, if the scene portrait is used for academic purposes, a formal and serious generation strategy should be selected; if the scene portrait is used for popular science scenarios, a lively and vivid generation strategy should be selected.

[0102] In this embodiment, there is a possibility that the target file does not meet the user's needs, and feedback should be provided for correction.

[0103] As an optional implementation, obtaining missing data marked for the target file;

[0104] The weight of the original data corresponding to the missing data is adjusted in the complex file and re-placed into the high-dimensional semantic space for mapping.

[0105] Here, manual correction is used to adjust the weight of the missing data, thereby optimizing the output target file.

[0106] It can be seen that through multimodal feature extraction and cross-modal attention alignment mechanism, multimodal data can be effectively extracted, and then semantic alignment and adaptive weight distribution can be achieved in high-dimensional semantic space, ensuring the consistency of multimodal information, eliminating semantic faults and information conflicts, and supplemented by structured knowledge drive. The ability to parse professional terminology, implicit logic and context-related content is greatly enhanced, the output quality is high, and it can be accurately output for the current application scenario, with a wide range of applications.

[0107] Example 2

[0108] See Figure. Figure 2 As shown, the complex document understanding system based on multimodal feature fusion and structured knowledge driving may include:

[0109] Feature extraction module, used to extract multimodal features for complex files;

[0110] Multimodal fusion module, used to map multimodal features in a high-dimensional semantic space and construct structured semantic data through a multi-head attention mechanism;

[0111] Knowledge fusion module, used to retrieve background information of structured semantic data using domain knowledge base;

[0112] The generation module is used to generate the target file based on the structured semantic data and the corresponding background information, and output the visual map to the user terminal according to the scene portrait and its generation strategy.

[0113] The above is a detailed introduction to a complex document understanding method and system driven by multimodal fusion and structured knowledge disclosed in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A complex document understanding method driven by multimodal fusion and structured knowledge, characterized by: The method comprises: Extract multimodal features for complex files; Mapping the multimodal features in a high-dimensional semantic space to construct structured semantic data; Retrieving background information of the structured semantic data using a domain knowledge base; A target file is generated based on the structured semantic data and corresponding background information.

2. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 1 is characterized in that: The multimodal feature extraction for complex files includes: Perform word segmentation, part-of-speech tagging, and stop word removal on text data to generate standardized characters; Identify the row and column structure, table headers, and data cells of the table data, and analyze the semantic associations between the data cells to establish a reference relationship between the table data and surrounding descriptions; Obtain multi-level visual features including coordinate axes, legends, and data points for chart data, and generate natural language descriptions; A segmentation network is used to locate the operators and scalars of formula data, and the corresponding syntax tree representation is generated based on the formal grammar.

3. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 1 is characterized in that: Mapping the multimodal features in a high-dimensional semantic space to construct structured semantic data includes: In the high-dimensional semantic space, a multi-head attention mechanism is used to calculate the correlation between different modal features to generate an inter-modal correlation matrix; Based on the association matrix, the contribution weight of each modal feature in the fusion process is dynamically adjusted through learnable weight parameters.

4. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 3 is characterized in that: The method further comprises: The calculation formula of the multi-head attention mechanism is as follows: Q=XW Q ,K=XW K ,V=XW V ; Among them, X is the multimodal splicing feature; dk represents the dimension scaling factor, where the square root of the feature dimension is taken; Matrix addition calculation is used to implement attention bias on the correlation matrix.

5. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 1 is characterized in that: The adopting of a domain knowledge base to retrieve background information of the structured semantic data includes: Identifying core topics and key entities in the structured semantic data, wherein the key entities include at least professional terms, organization names, and specific concepts; Extracting semantic associations between the core topics and the key entities through a relationship extraction algorithm, including explicit annotation relationships and implicit relationships based on contextual reasoning; Constructing a query instruction based on the core topic and the key entity; Based on the query instruction, a targeted search is performed in the domain knowledge base to obtain background information including knowledge graph fragments, background knowledge, industry standards, professional term definitions or domain-specific rules.

6. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 5 is characterized in that: The semantic association between the core topic and the key entity is extracted by the relationship extraction algorithm, including explicit annotation relationship and implicit relationship based on contextual reasoning, including: A graph neural network is used to transmit messages and update node representations on the knowledge graph composed of the core topics and key entities, thereby realizing dynamic knowledge reasoning based on the graph structure. Wherein, the dynamic knowledge reasoning integrates contextual semantic information.

7. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 6 is characterized in that: The generating of a target file based on the structured semantic data and the corresponding background information includes: Building a sequence-to-sequence model based on the Transformer encoder; Using the explicit or implicit relationships obtained by the graph neural network reasoning as joint input to obtain high-weight information; Combining the structured semantic data and enhanced feature representation, a target file that meets domain logic and specification requirements is generated.

8. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 7 is characterized in that: The method further comprises: Preset a number of scene portraits, each of which has a corresponding generation strategy; The generation strategy includes professional depth weight, information strength control rules, language style template and user interaction protocol, and the generation strategy of different scene portraits has differentiated settings; According to the selected scenario portrait, the target file is optimized and output based on its corresponding generation strategy.

9. The method for understanding complex documents driven by multimodal fusion and structured knowledge according to claim 1 is characterized in that: The method further comprises: Obtaining missing data marked for the target file; The weight of the original data corresponding to the missing data is adjusted in the complex file, and is re-placed into the high-dimensional semantic space for mapping.

10. A complex document understanding system driven by multi-modal fusion and structured knowledge, characterized by: include: Feature extraction module, used to extract multimodal features for complex files; A multimodal fusion module is used to map the multimodal features in a high-dimensional semantic space and construct structured semantic data through a multi-head attention mechanism; A knowledge fusion module, configured to retrieve background information of the structured semantic data using a domain knowledge base; The generation module is used to generate a target file based on the structured semantic data and the corresponding background information, and output a visual map to the user terminal according to the scene portrait and its generation strategy.

Images