Knowledge graph multi-mode document analysis and image table semantization knowledge recall method

Abstract

The invention discloses a knowledge graph multi-modal document analysis and image table semantization knowledge recall method, and belongs to the technical field of knowledge engineering and information retrieval. The invention provides an innovative scheme for fusing a visual language model, semantic abstract generation and knowledge graph modeling. The method comprises the following steps: constructing a vertical domain knowledge graph by adopting a BERT-BiLSTM-CRF model; according to the method, multi-modal document analysis is realized through models such as DocLayout-YOLO, TableMaster, UniMERNet and the like; the method comprises the following steps of: segmenting an image into 16 * 16 block sequences by adopting a vit-gpt2-image-adaptation model, and realizing image semantization through 768-dimensional vector space mapping and Transform coding; constructing a document summary tree based on DBSCAN clustering and LLM recursive summary; and designing a hybrid retrieval space fusing semantic vectors and structured vectors, and reordering by adopting a double-attention mechanism. According to the method, the knowledge base document retrieval recall rate is increased to 99%, the question and answer accuracy rate reaches 90% or above, the index construction time is shortened by 60%, and the problem that semantic understanding and recall of non-text elements in complex documents are difficult is effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of knowledge engineering and information retrieval technology, and in particular to methods for knowledge graph multimodal document parsing and image-table semantic knowledge retrieval. Background Technology

[0002] With the rapid development of the digital age, enterprises and institutions have accumulated a large number of complex documents containing multimodal content such as text, images, tables, and formulas. Traditional document processing methods mainly rely on OCR technology for text recognition, but they have significant limitations in handling complex layouts, table structures, mathematical formulas, and image semantic understanding. Existing document parsing technologies often adopt a single-modal processing approach, which cannot effectively integrate the multiple information types in a document, leading to the loss of key information and inaccurate semantic understanding. At the same time, traditional keyword-matching-based retrieval methods struggle to understand the deep semantics of documents and cannot meet complex query needs.

[0003] In the field of knowledge management, while knowledge graph technology provides an effective solution for structured knowledge storage and reasoning, existing knowledge graph construction mainly targets plain text data and lacks the ability to effectively process multimodal document content. Existing methods often ignore the rich semantic information contained in non-text elements such as images and tables during knowledge extraction, resulting in incomplete and inaccurate knowledge graphs. Furthermore, traditional knowledge graph update mechanisms are mostly batch-based full updates, which cannot cope with dynamically changing knowledge needs, leading to problems such as poor timeliness and high maintenance costs in practical applications.

[0004] In information retrieval, traditional retrieval systems primarily rely on word frequency statistics and simple semantic matching, making it difficult to handle complex reasoning queries and cross-modal information fusion. While RAG (Retrieval Augmentation) technology, which has emerged in recent years, has improved retrieval performance to some extent, existing methods still have shortcomings in handling domain-specific knowledge, complex document structures, and multimodal information fusion. Particularly when faced with technical documents containing specialized charts, complex tables, and mathematical formulas, existing retrieval systems often fail to accurately understand and recall relevant knowledge, severely impacting the quality of knowledge services and user experience. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problems in the prior art, such as inaccurate multimodal document parsing, incomplete knowledge graph construction, difficulty in cross-modal information fusion, and poor knowledge retrieval and recall effects, and to provide a knowledge recall method that integrates vertical domain knowledge graphs with multimodal document parsing and image and table semantics.

[0006] A knowledge graph multimodal document parsing and image-table semantic knowledge retrieval method includes: In the knowledge preprocessing stage, a vertical domain knowledge graph is constructed. The BERT-bidirectional LSTM-CRF nested entity recognition model is used to extract domain entity relationship triples. A concept hierarchy system is constructed based on SKOS and entity attribute constraints are defined using the OWL ontology language to form a schema specification. Knowledge is stored through the Neo4j attribute graph model and the Diff-Patch algorithm is used for graph version management. In the multimodal document parsing stage, document elements are located using DocLayout-YOLO and LayoutLMv3 models, formula positions are located using the YOLOv8 model, formula images are converted into LaTeX code using the UniMERNet model, text content is recognized using PaddleOCR, and table structure is parsed using the TableMaster model. In the image semanticization stage, the nlpconnect / vit-gpt2-image-captioning model is used to segment the input image into a sequence of equal-sized blocks with a resolution of 16×16 and perform linear embedding processing. The TransformerEncoder structure is used to achieve cross-fusion of image block features. Based on BPE word segmentation technology and pixel-level segmentation mask annotation, the mapping relationship between local regions of the image and text semantics is established. In the knowledge retrieval and recall stage, semantic relationships in the knowledge graph are mined through graph neural networks to generate a set of similar questions. A hybrid embedding model based on the semantic vector space of Transformer and the structured vector space based on the knowledge graph is constructed. A dual attention mechanism re-ranking model is used to fuse question attention and knowledge attention to rank the retrieval results.

[0007] Furthermore, in the construction of the vertical domain knowledge graph, an incremental knowledge fusion based on time window sliding is adopted. New knowledge triples are collected within each time window T, and new knowledge is integrated into the existing graph through entity alignment and relation disambiguation algorithms. A dual interface of SPARQL endpoint and RESTful API is designed. The SPARQL interface supports RDF query, and the RESTful interface provides JSON format query.

[0008] Furthermore, the process of constructing a document summary tree after multimodal document parsing includes: spatial clustering based on the positional relationship and content similarity of document blocks using the DBSCAN algorithm; inputting ordered document blocks into a large language model in groups of g for content summarization; using the generated summaries as new nodes to continue recursively summarizing in groups of g to form a hierarchical summary tree; using SpaCy to process each document block to extract entities and nouns; constructing connection edges between entities based on sentence co-occurrence relationships; and merging all subgraphs to form a global entity relationship graph.

[0009] Furthermore, in the image semantic processing, each image patch is mapped to a 768-dimensional vector space through a linear projection layer, and a learnable position embedding vector is incorporated to represent the spatial location. Feature extraction is performed through a TransformerEncoder structure containing 12 layers of multi-head self-attention mechanism, with each layer containing 12 attention heads. In the Encoder-DecoderSelf-Attention layer, the image encoder output is used as a key-value pair, and the hidden state of the text decoder is used as a query. The alignment of visual and linguistic features is achieved through a cross-attention mechanism.

[0010] Furthermore, the problem handling in knowledge retrieval recall includes: identifying the subject-verb-object structure of the problem through dependency parsing, extracting key entities using named entity recognition, and determining the query type using an intent recognition model; and comprehensively determining the mapping relationship between sub-problems and knowledge graph concept nodes using three methods: string matching based on edit distance, semantic similarity calculation based on Word2Vec, and relevance scoring based on graph path.

[0011] Furthermore, in the hybrid embedding model, the semantic vector space uses BERT or Sentence-BERT models to generate 768-dimensional dense vector representations, while the structured vector space generates vector representations of entities and relations through TransE or ComplEx knowledge graph embedding methods. The vector representation capability is enhanced through a contrastive learning strategy using the InfoNCE loss function, and a multi-layer hash table is constructed using the LSH algorithm to achieve fast approximate retrieval of high-dimensional vectors.

[0012] Furthermore, in the dual-attention mechanism re-ranking model, question attention evaluates the relevance of a document to a question by calculating the attention score of the dot product of the query vector and the document vector, while knowledge attention calculates attention weights based on the centrality and authority indicators of a document in the knowledge graph; the REINFORCE algorithm is used to optimize the ranking strategy based on user click feedback.

[0013] Furthermore, the final generation of knowledge retrieval recall includes: designing an adaptive prompt word generation template based on the question type, retrieval result characteristics, and target application scenario; dynamically combining question descriptions, knowledge fragments, and guiding instructions; inputting the reordered retrieval results and prompt words into a large language model for knowledge integration and rewriting; and verifying the consistency between entity relationships in the generated content and the knowledge graph through triple matching.

[0014] To achieve the above objectives, this invention provides a knowledge graph multimodal document parsing and image table semantic knowledge retrieval method, comprising two core stages: knowledge preprocessing and retrieval. In the knowledge preprocessing stage, a BERT-bidirectional LSTM-CRF nested entity recognition model is used to extract domain entity relationship triples. A concept hierarchy system is constructed based on SKOS, and entity attribute constraints are defined using the OWL ontology language. Knowledge is stored through the Neo4j attribute graph model, and incremental graph version management is performed using the Diff-Patch algorithm. For multimodal document parsing, the DocLayout-YOLO and LayoutLMv3 models are integrated for document element localization. Combined with YOLOv8 formula localization, UniMERNet formula recognition, PaddleOCR text recognition, and TableMaster table parsing technologies, accurate parsing of complex documents is achieved. Image semantic processing employs the nlpconnect / vit-gpt2-image-captioning model, using a combined architecture of VisionTransformer and GPT-2 to convert image content into understandable text descriptions. It then establishes a precise mapping between local image regions and text semantics based on BPE word segmentation technology combined with pixel-level segmentation masks. In the retrieval and recall stage, graph neural networks are used to mine latent semantic relationships in the knowledge graph, constructing a hybrid embedding model that integrates semantic and structured vector spaces. A dual-attention mechanism re-ranking model optimizes the ranking of retrieval results. Finally, high-quality knowledge integration and generation are achieved through adaptive prompt word generation and a large language model.

[0015] First, by integrating multiple advanced deep learning models, this invention achieves accurate parsing of complex documents containing multimodal content such as text, images, tables, and formulas, significantly improving the accuracy and completeness of information extraction compared to traditional methods. Second, it innovatively constructs a vertical domain knowledge graph based on incremental updates, and combines it with knowledge fusion technology using a sliding time window, effectively solving the dynamic maintenance problem of the knowledge graph and improving update efficiency by 40% compared to traditional static graphs. Third, it proposes an innovative scheme for image semantic processing, achieving semantic conversion of "images as text" through deep fusion of VisionTransformer and GPT-2, providing effective technical support for the unified processing of multimodal information. Fourth, it designs a knowledge retrieval and recall mechanism based on advanced RAG, achieving a 99% knowledge recall rate and over 90% question-answering accuracy in financial document processing scenarios through hybrid embedding models and dual attention reordering, reducing index construction time by 60% compared to traditional methods, fully demonstrating the superiority and practical value of the method presented in this invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the knowledge retrieval method of the present invention, showing the complete process from inputting a document to outputting an answer; Figure 2 The flowchart for constructing the vertical domain knowledge graph of the present invention illustrates the processing flow from raw data to graph storage and consumption. Figure 3 This is a flowchart of the image table semantic processing of the present invention, illustrating the image-to-text conversion process; Figure 4 The flowchart of the multimodal document parsing of the present invention illustrates the document layout recognition and element parsing process; Figure 5 The flowchart for constructing the document summary tree of this invention illustrates the process of clustering and grouping document blocks and recursively summarizing them to form a hierarchical tree. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0019] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0020] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0021] See Figures 1 to 5 As shown This invention provides a complex knowledge retrieval method that integrates vertical domain knowledge graphs with multimodal document parsing and image table semantics. The method includes two core stages: knowledge preprocessing and retrieval retrieval. It is an innovative three-in-one solution that integrates visual language models to achieve accurate OCR-free parsing, semantic summary generation to deeply extract the core information of non-text elements, and knowledge graph modeling of entities and relationships to enhance retrieval.

[0022] Implementation Method 1: Construction of Vertical Domain Knowledge Graph In the knowledge extraction stage, this invention employs a "domain entity-relation triple extraction" strategy. For semi-structured and unstructured industry data, it constructs a nested entity recognition model based on BERT-bidirectional LSTM-CRF, combined with a rule engine to achieve named entity recognition, relation extraction, and event extraction. Specifically, the input industry text data is first preprocessed, including basic processing such as word segmentation and part-of-speech tagging. Then, the processed text is input into the BERT model to obtain context-related word vector representations. Next, a bidirectional LSTM network captures the forward and backward dependencies of the sequence. Finally, a CRF layer is used for sequence labeling to identify entities and their types in the text. For relation extraction, a method combining remote supervision and rule templates is used to match the semantic associations between entity pairs through predefined relation patterns.

[0023] In the graph generation phase, an industry-specific conceptual hierarchy is constructed based on SKOS (Simple Knowledge Organization System), and entity attribute constraints are defined using the OWL (Web Ontology Language) ontology language, forming a domain-specific schema specification. The Neo4j attribute graph model is used to store knowledge, where nodes represent entities, edges represent relationships, and attributes store specific information about entities and relationships. Semantic association retrieval is achieved through the Cypher query language, supporting complex graph traversal and pattern matching operations. A Diff-Patch algorithm is introduced for graph version management. This algorithm generates incremental update patches by calculating the differences between the old and new graphs, avoiding the performance overhead of full updates. An incremental knowledge fusion technology based on a sliding time window is designed. A time window of T is set, and newly added knowledge triples are collected within each window. New knowledge is integrated into the existing graph through entity alignment and relationship disambiguation algorithms. Taking the maintenance of financial policies and regulations as an example, the efficiency of updating the graph is improved by 40% compared to traditional static graphs, effectively eliminating knowledge conflicts and ensuring timeliness.

[0024] At the graph consumption level, a reverse design driven by application scenarios is adopted. API call logs from consumption scenarios are analyzed to statistically determine the frequency and response time of various query patterns, dynamically adjusting relationship weight parameters in the schema. A dual interface is designed: a SPARQL endpoint and a RESTful API. The SPARQL interface supports standard RDF queries, while the RESTful interface provides simplified JSON format queries. Semantic reasoning services based on a rule engine are supported, employing a hybrid reasoning strategy combining forward and backward chaining to achieve real-time knowledge graph queries in complex document retrieval scenarios. Inference latency is controlled within 50ms, supporting second-level association queries for 100,000 entities.

[0025] Implementation Method 2: Multimodal Document Parsing and Document Summary Tree Construction This method innovatively integrates multimodal parsing technologies such as text parsing, OCR recognition, table recognition, formula recognition, and layout recognition. In the multimodal parsing stage, the DocLayout-YOLO and LayoutLMv3 models are used to accurately locate document elements. DocLayout-YOLO is responsible for document layout analysis, identifying areas such as titles, paragraphs, images, and tables, while LayoutLMv3 further understands the semantic structure of the document. The YOLOv8 model is specifically responsible for locating formulas (inline and block formulas). Through training on a large amount of mathematical document data, the model can accurately recognize the location of mathematical expressions in LaTeX format. The UniMERNet model performs formula recognition, converting the located formula images into editable LaTeX code. Text content is recognized using PaddleOCR, an end-to-end OCR system employing DB text detection and CRNN text recognition, supporting high-precision recognition of multilingual text. Table structures are parsed by the TableMaster model, which can recognize complex layouts such as row and column structures and merged cells, and outputs the table content in a structured manner.

[0026] For the parsed data, a hierarchical processing strategy is adopted for secondary knowledge processing. First, based on semantic similarity and spatial layout features, document blocks are clustered. The DBSCAN algorithm is used to perform spatial clustering based on the positional relationship and content similarity of document blocks, forming semantically related document block groups. Then, a document summary tree is recursively constructed. Ordered document blocks are input into a large language model in groups of g (g is a configurable parameter, defaulting to 5) for content summarization. To optimize resource consumption, overlapping content of adjacent blocks is merged. The generated summary is used as a new node, and recursively summarizing continues in groups of g, ultimately forming a hierarchical summary tree. The root node contains global summary information of the document, and the leaf nodes contain specific and detailed original content. Simultaneously, SpaCy is used to process each document block, extracting entities and nouns. Connection edges between entities are constructed based on intra-sentence co-occurrence relationships. That is, if two entities appear in the same sentence, an association edge is established between them, and the weight of the edge is determined by the co-occurrence frequency. All subgraphs are merged to form a global entity relationship graph covering the entire text, which reflects the semantic association network between various concepts in the document.

[0027] Implementation Method 3: Semantic Image and Table Processing This method achieves cross-modal extension from image caption generation to optical character recognition, innovatively constructing an integrated technical path for image feature extraction and text generation. It employs the nlpconnect / vit-gpt2-image-captioning model for image semantic processing, based on a combined architecture of VisionTransformer (ViT) and GPT-2. The input image is segmented into a sequence of 16×16 resolution equal-sized blocks, with each block consisting of 14×14 patches of a 224×224 pixel image. Linear embedding is performed on each image block, mapping each patch to a 768-dimensional vector space through a linear projection layer, while incorporating positional information encoding. A learnable positional embedding vector represents the spatial location of each patch within the image.

[0028] The standard TransformerEncoder architecture incorporates a 12-layer multi-head self-attention mechanism, with 12 attention heads per layer, enabling cross-fusion and deep extraction of features from various image patches. For text processing, the text is tokenized and decomposed into sub-word units using BPE (BytePairEncoding) segmentation. Each text token is sequentially processed through an embedding layer to obtain word vector representations, followed by feature transformation through a MaskedSelf-Attention layer to ensure autoregressive characteristics in the generation process. Feature fusion is then performed between the Encoder-DecoderSelf-Attention layer and the TransformerEncoder output. The image encoder output serves as the key-value pair, and the hidden state of the text decoder serves as the query, achieving alignment of visual and linguistic features through a cross-attention mechanism.

[0029] Based on BPE word segmentation technology combined with pixel-level segmentation mask annotation, a precise mapping relationship between local regions of an image and text semantics is established. In practice, Mask R-CNN or SAM (SegmentAnythingModel) is used to generate segmentation masks for the image, and each masked region is associated with specific words in the generated descriptive text, thereby achieving semantic conversion of "image as text." This technology provides key technical support for multimodal large-scale models to process complex structures such as characters, formulas, and tables, effectively improving the understanding and interaction capabilities of multimodal information.

[0030] Implementation Method 4: Knowledge Retrieval and Recall Based on Advanced RAG This method proposes a knowledge retrieval recall approach based on advanced RAG (Retrieval Enhanced Generation). In the initial stage of question processing, a strategy combining semantic parsing and syntactic analysis is employed to deeply decompose user questions. Dependency parsing identifies the subject-verb-object structure of the question, named entity recognition technology extracts key entities from the question, and an intent recognition model determines the query type. By introducing a dynamic ontology mapping mechanism, the decomposed sub-questions are associated with concept nodes in the vertical domain knowledge graph in multiple dimensions. Specifically, three methods are used to comprehensively determine the mapping relationship: string matching based on edit distance, semantic similarity calculation based on Word2Vec, and relevance scoring based on graph path.

[0031] This study utilizes Graph Neural Networks (GNNs) to mine latent semantic relationships within a graph. The GraphSAGE algorithm is employed to learn the representations of graph nodes, updating node representations by aggregating feature information from neighboring nodes. This generates a set of questions semantically similar to the user's question, providing rich candidate clues for subsequent retrieval. Based on the logical correlation and semantic coherence of question elements, reinforcement learning algorithms are used to filter and recombine similar questions. The DeepQ-Network algorithm is employed, modeling the question recombination process as a Markov decision process. By maximizing the expected reward, the question combination strategy is optimized to form structurally complete and semantically clear assembled questions.

[0032] In the vectorized retrieval stage, a hybrid embedding model is constructed, mapping the assembly problem simultaneously to a Transformer-based semantic vector space and a knowledge graph-based structured vector space. The semantic vector space uses BERT or Sentence-BERT models to generate 768-dimensional dense vector representations, while the structured vector space generates entity and relation vector representations through knowledge graph embedding methods such as TransE or ComplEx. A contrastive learning strategy is employed, using the InfoNCE loss function to enhance the vector representations' ability to capture semantic and structural information. Hierarchical hash indexing technology is utilized, employing the LSH algorithm to construct a multi-layered hash table, enabling fast approximate retrieval of high-dimensional vectors, significantly improving retrieval efficiency while maintaining retrieval accuracy.

[0033] To address large-scale retrieval results, a dual-attention re-ranking model is proposed, fusing question attention and knowledge attention. Question attention evaluates the relevance of documents to the question by calculating the attention score of the dot product between the query vector and the document vector; knowledge attention calculates attention weights based on indicators such as the centrality and authority of documents in the knowledge graph. Reinforcement learning is introduced to dynamically adjust the ranking weights, and the REINFORCE algorithm is used to optimize the ranking strategy based on user click feedback, ensuring that the most relevant knowledge is ranked first.

[0034] Finally, an adaptive prompt generation template is designed to dynamically combine question descriptions, knowledge fragments, and guiding instructions based on user question type (factual, inferential, comparative, etc.), retrieval result characteristics (text, tables, images, etc.), and target application scenario (question-answering, summarizing, translation, etc.). The reordered retrieval results and optimized prompts are then input into a large language model, leveraging the model's generation capabilities to deeply integrate and rewrite the retrieved knowledge. The generated content undergoes logical verification and fact-checking using structured information from the knowledge graph. Triple matching verifies whether the entity relationships in the generated content are consistent with the knowledge graph, outputting high-quality answers that meet user needs, thus achieving closed-loop optimization from retrieval to generation. Example

[0035] Taking the processing of policy and regulatory documents in the financial field as an example, the specific implementation process of the method of this invention is explained in detail. Assume that a batch of financial regulatory documents containing text, tables, formulas, and charts needs to be processed, and a knowledge base system supporting complex queries needs to be built.

[0036] First, a vertical domain knowledge graph is constructed. Knowledge of financial terminology, regulatory agencies, regulations, and financial products is collected. The BERT-BiLSTM-CRF model is used to identify professional entities such as "People's Bank of China," "reserve requirement ratio," and "liquidity coverage ratio." Specific relationships in areas such as "regulator-regulated," "inclusion-included," and "revision-revision" are identified through relation extraction. The extracted knowledge is stored in the Neo4j graph database, forming a financial knowledge graph containing 500,000 entity nodes and 2 million relation edges.

[0037] Next, multimodal document parsing is performed. For a bank regulatory report containing complex tables, DocLayout-YOLO is used to identify 3 heading areas, 15 paragraph areas, 8 table areas, and 5 formula areas in the document. For the identified table areas, the TableMaster model is used to parse the row and column structure of the tables. For example, a table showing "Capital Adequacy Ratio Requirements for Various Financial Institutions" is parsed into 5 rows and 8 columns of structured data. For mathematical formulas such as "CAR=(T1+T2) / RWA×100%", YOLOv8 is used to locate the formula position, and UniMERNet recognizes and converts it into a computable LaTeX format.

[0038] Subsequently, a document summary tree was constructed. The parsed document blocks were grouped into GPT-4 for summarization in sets of five. The first level generated three summary nodes, summarizing "Regulatory Background," "Specific Requirements," and "Implementation Rules," respectively. The second level further summarized these three summaries, forming a root node containing the core points of the entire document: "This document specifies the calculation method and regulatory requirements for the capital adequacy ratio of commercial banks, with a focus on the additional capital requirements for systemically important banks." Simultaneously, SpaCy was used to extract key entities such as "commercial banks," "capital adequacy ratio," and "risk-weighted assets," constructing an entity relationship graph.

[0039] For the regulatory framework diagrams in the document, image semantic processing is performed. The diagrams are input into the vit-gpt2-image-captioning model to generate a semantic description: "This diagram illustrates a three-tiered regulatory framework, with the People's Bank of China at the top, the State Financial Regulatory Commission in the middle, and various commercial banks at the bottom. Arrows indicate the transmission of regulatory relationships." Through pixel-level segmentation, a mapping relationship is established between each institution's logo area and its corresponding text description.

[0040] Finally, knowledge retrieval and recall are implemented. When a user queries "What are the capital adequacy requirements for systemically important banks?", the system first breaks the question down into two subqueries: "systemically important banks" and "capital adequacy requirements". Relevant concepts such as "D-SIBs", "additional capital requirements", and "countercyclical capital buffer" are found through a knowledge graph. The expanded query is vectorized, and five relevant document blocks are retrieved from the vector library. Through dual-attention reordering, the table content containing specific numerical requirements is placed first. The final answer is generated: "According to the latest regulatory requirements, the capital adequacy requirements for systemically important banks are above the basic 8%, and an additional capital requirement of 1-3.5% is also required. The specific percentage is determined based on the bank's systemic importance score." Through the above implementation, the method of the present invention achieves a knowledge recall rate of 99% and a question-answering accuracy rate of over 90% in financial document processing scenarios, and the index construction time is reduced by 60% compared with traditional methods, fully verifying the effectiveness and practicality of the method.

[0041] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0042] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A knowledge graph multi-modal document parsing, image table semanticization knowledge recall method, characterized in that, Comprise: The knowledge preprocessing stage, construct the vertical knowledge graph, adopt the BERT-bidirectional LSTM-CRF nested entity recognition model to extract the domain entity relationship triplets, construct the concept hierarchy system based on SKOS and define the entity attribute constraints to form the Schema specification using the OWL ontology language, store the knowledge through the Neo4j attribute graph model and use the Diff-Patch algorithm for graph version management; The multi-modal document parsing stage, locate the document elements through the DocLayout-YOLO and LayoutLMv3 models, locate the formula position through the YOLOv8 model, convert the formula image to LaTeX code through the UniMERNet model, recognize the text content through the PaddleOCR, and parse the table structure through the TableMaster model; The image semanticization stage, adopt the nlpconnect / vit-gpt2-image-captioning model to divide the input picture into 16x16 resolution equal size block sequences and perform linear embedding processing, realize the cross-fusion of picture block features through the TransformerEncoder structure, and establish the mapping relationship between the local area of the picture and the text semantics based on the BPE word segmentation technology combined with the pixel-level segmentation mask annotation; The knowledge retrieval and recall stage, generate a set of similar questions by mining the semantic relationships in the knowledge graph through the graph neural network, construct a hybrid embedding model based on the semantic vector space of the Transformer and the structured vector space of the knowledge graph, and use a double attention mechanism reordering model to fuse problem attention and knowledge attention to sort the retrieval results.

2. The method of claim 1, wherein, In the construction of the vertical knowledge graph, incremental knowledge fusion based on time window sliding is adopted, new knowledge triplets are collected within each time window T, and new knowledge is integrated into the existing graph through entity alignment and relationship disambiguation algorithms; Design SPARQL endpoint and RESTful API dual interface, SPARQL interface supports RDF query, and RESTful interface provides JSON format query.

3. The method of claim 1, wherein, The process of building a document summary tree after multi-modal document parsing includes: based on the DBSCAN algorithm, the spatial clustering is performed according to the position relationship and content similarity of the document blocks, the ordered document blocks are input into the large language model in groups of g, the generated abstract is used as a new node to continue recursive summarization in groups of g to form a hierarchical abstract tree; Use SpaCy to process each document block to extract entities and nouns, construct the connection edges between entities based on intra-sentence co-occurrence relationship, and fuse all subgraphs to form a global entity relationship graph.

4. The method of claim 1, wherein, In the image semantic processing, each picture block is mapped to a 768-dimensional vector space through a linear projection layer, and a learnable position embedding vector is used to represent the space position; a TransformerEncoder structure containing 12 layers of multi-head self-attention mechanism is used for feature extraction, and each layer contains 12 attention heads; in the Encoder-Decoder Self-Attention layer, the image encoder output is used as the key-value pair, and the text decoder hidden state is used as the query, and the cross-attention mechanism is used to realize the alignment of visual and language features.

5. The method of claim 1, wherein, The problem processing in the knowledge retrieval recall includes: identifying the subject-predicate-object structure of the question through dependency syntax analysis, extracting key entities using named entity recognition, and determining the query type using an intent recognition model; three methods of string matching based on edit distance, semantic similarity calculation based on Word2Vec, and correlation score based on graph path are used to determine the mapping relationship between sub-questions and knowledge graph concept nodes.

6. The method of claim 1, wherein, In the mixed embedding model, the semantic vector space uses the BERT or Sentence-BERT model to generate a 768-dimensional dense vector representation, and the structured vector space uses the TransE or ComplEx knowledge graph embedding method to generate vector representations of entities and relationships; the vector representation ability is enhanced through the contrast learning strategy of the InfoNCE loss function, and the LSH algorithm is used to construct a multi-layer hash table to realize fast approximate retrieval of high-dimensional vectors.

7. The method of claim 1, wherein, In the double attention mechanism reordering model, the problem attention evaluates the relevance of the document and the question by calculating the dot product attention score of the query vector and the document vector, and the knowledge attention calculates the attention weight based on the centrality and authority indicators of the document in the knowledge graph; The REINFORCE algorithm is used to optimize the ranking strategy according to user click feedback.

8. The method of claim 1, wherein, The final generation of the knowledge retrieval recall includes: designing an adaptive prompt word generation template according to the problem type, retrieval result features, and target application scenario, dynamically combining question descriptions, knowledge fragments, and guidance instructions; inputting the reordered retrieval results and prompt words into a large language model for knowledge integration and rewriting, and verifying the consistency of entity relationships in the generated content with the knowledge graph through triple matching.

Images