Industry standard document oriented deep semantic entity and relation automatic extraction method

Abstract

The application discloses an automatic extraction method for deep semantic entities and relations of an industry standard document, relates to the technical field of intelligent document processing, and comprises the following steps: inputting a document page image into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph; processing a transformer architecture model to obtain semantic entities; generating a training data set by using a multi-level noise reduction neural network, extracting the heterogeneous graph, and obtaining the heterogeneous graph; fusing an edge-oriented graph attention network model to obtain the relation result of the heterogeneous graph entity recognition; processing the multi-modal document heterogeneous graph, the semantic entities, the heterogeneous graph and the relation result of the heterogeneous graph entity recognition by using a coreference resolution model, combining a training link prediction model for calculation, and obtaining a global knowledge network. The application provides a more pure and reliable training data set, and improves the extraction accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent document processing, in particular to a deep semantic entity and relationship automatic extraction method for industry standard documents. BACKGROUND

[0002] Industry standard documents are important technical documents that guide industrial production, guarantee product quality, and promote industrial development. The documents are usually presented in PDF format and contain a large amount of structured and semi-structured knowledge, such as technical requirements, performance indicators, test methods, and equipment parameters. Efficient and accurate extraction of these knowledge from the documents and construction of a machine-readable knowledge graph have great value for intelligent question answering, compliance checking, technical research and development, and knowledge management, and are located in the fields of artificial intelligence, natural language processing, knowledge graph construction, and document intelligence processing. However, the existing technology faces the following severe challenges when processing such documents: difficulty in cross-document knowledge association, large industry standard system, and a large number of reference and relationship between different standards. The existing methods are difficult to automatically identify these cross-document references and relationships, resulting in isolated knowledge and inability to construct a complete industry knowledge network.

[0003] Low efficiency and poor generalization in domain adaptation. Although pre-training language models represented by BERT have achieved remarkable results in natural language processing tasks, when applied to highly specialized industry standard fields, complete model fine-tuning is usually required, and a large amount of domain annotation data, high computing cost and storage overhead are required. For each new industry or new standard system, a repeated high-cost process is required, which limits the scalability and economy of the technical solution. At the same time, in the case of scarce annotation data, full fine-tuning can easily cause the model to overfit in a specific domain, losing the ability to generalize.

[0004] In addition, existing entity and relationship extraction methods rely heavily on surface text features and are difficult to understand the deep semantics in standard documents, such as the inability to distinguish between different types of parameter entities and the inability to identify the implicit relationship between test methods and test results. The noise problem in weakly supervised relationship extraction is prominent. To reduce the dependence on manual annotation, distant supervision (Distant Supervision) or weak supervision methods are widely used in relationship extraction. The core assumption is that if there is a relationship r between entity pair (e1, e2) in the knowledge base, all sentences containing both entity mentions express relationship r. However, this assumption is too strict in practice and will introduce a lot of noise labels. For example, a sentence may mention both the insulation resistance and the test current, but describe two different parameters respectively, and does not express the test condition relationship between them. Traditional methods require a large amount of manual annotation to train the model, and the standard document field is strong, which not only has a high annotation cost, but also requires annotators to have deep domain knowledge, resulting in low annotation efficiency.

[0005] The modeling capability for long-distance and implicit relationships is insufficient, the entity relationship in the industry standard often spans multiple sentences, paragraphs or even chapters, for example, the technical parameters defined in the first chapter may be detailed in a table in the fifth chapter, the existing relationship extraction model, such as the end-to-end model based on LSTM or Transformer (i.e. deep learning model in the field of artificial intelligence), although it can handle intra-sentence relationships, but it has limited capability in modeling such long-distance and cross-structure complex dependency relationships, multiple models usually treat the document as a flat sequence or a simple graph structure, it is difficult to capture the deep semantic association determined by the document logical hierarchy such as chapter-paragraph-list and spatial layout and table rows and columns, and traditional methods often treat PDF documents as linear text streams, ignoring rich visual formatting information. There are a large number of tables, nested lists, charts, mathematical formulas and column layouts in industry standard documents, and simple text extraction techniques are difficult to handle these complex two-dimensional structures, which easily leads to loss of key information or structural disorder, for example, it cannot correctly associate table headers with cell contents, or it cannot identify the hierarchical relationship of nested lists, so that the subsequent information extraction task has the problem of incomplete information from the source, and cannot realize unified and structured storage of information.

[0006] The existing technology has a series of problems such as shallow format understanding, high domain adaptation cost, large noise interference of weak supervision, and weak complex relationship modeling capability when dealing with industry standards and other complex documents, and it is difficult to realize automatic, high-precision and efficient knowledge extraction.

[0007] At present, no effective solutions have been proposed for the problems in the related art. SUMMARY

[0008] In view of the problems in the related art, the present application proposes a deep semantic entity and relationship automatic extraction method for industry standard documents to overcome the above technical problems existing in the prior art.

[0009] To this end, the specific technical solutions adopted by the present application are as follows:

[0010] The present application proposes a deep semantic entity and relationship automatic extraction method for industry standard documents, comprising:

[0011] S1, input the document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph;

[0012] S2, process the multi-modal document heterogeneous graph and the preset entity type through a transformer architecture model to extract the embedding vectors of the multi-modal document heterogeneous graph and the preset entity type, and obtain the semantic entities in the document;

[0013] S3, generating a training data set by using a multi-stage noise reduction neural network on the document corpus to be processed and a pre-configured knowledge base, extracting an heterogeneous graph from the training data set and the semantic entity in the document respectively, and obtaining the heterogeneous graph;

[0014] S4, fusing the heterogeneous graph by using an edge-oriented graph attention network model, obtaining a relationship representation vector of the heterogeneous graph, and performing relationship probability distribution on the relationship representation vector of the heterogeneous graph by using a multi-layer perception classifier, and obtaining a relationship result of the heterogeneous graph entity recognition;

[0015] S5, processing the multi-modal document heterogeneous graph, the semantic entity, the heterogeneous graph, and the relationship result of the heterogeneous graph entity recognition in sequence by using a co-reference resolution model, and performing calculation and threshold determination by combining a pre-trained link prediction model, and obtaining a global knowledge network.

[0016] Further, the processing of the document page image to be processed by inputting the document page image to the multi-modal document understanding model to obtain the multi-modal document heterogeneous graph comprises:

[0017] S11, inputting the document page image to be processed into the multi-modal document understanding model by integrating text and image processing capabilities to recognize all text blocks and visual elements on the page, and obtaining a processed document page image;

[0018] S12, recognizing the processed document page image by using a full convolutional network to obtain a document page chart, and combining structure recognition and content association to obtain structure data of the document page graph;

[0019] S13, performing semantic parsing on the structure data by using an attention mechanism encoder-decoder model and a context-free grammar parser to obtain a semantic representation of the structure data;

[0020] S14, outputting and integrating the processed document page image, the document page chart, the structure data of the document page graph, and the semantic representation of the structure data to construct a multi-modal document heterogeneous graph.

[0021] Further, the multi-modal document understanding model comprises:

[0022] extracting text and functions for classification, and pre-training on a large amount of documents, the multi-modal document understanding model predicting a logical role for each text block;

[0023] The logical roles include title, chapter title, footnote, header, footer, page number, and list item.

[0024] Further, the multi-modal document heterogeneous graph and the preset entity type are processed through the transformer architecture model to extract embedding vectors of the multi-modal document heterogeneous graph and the preset entity type, and semantic entities in the document are obtained.

[0025] S21, the multi-modal document heterogeneous graph and the preset entity type are taken as core input data, embedding vectors of the multi-modal document heterogeneous graph and the preset entity type are extracted through standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding and boundary box coordinate addition calculation;

[0026] S22, the extracted embedding vectors are sequentially processed through the low-rank adapter and the regularization technique to obtain semantic entities in the document.

[0027] Further, the regularization technique includes:

[0028] attention dropout and weight decay;

[0029] Among them, the attention dropout inputs the extracted embedding vectors into the attention calculation to obtain attention scores, selects part of the attention scores to be zero through random selection, and inputs the attention scores into a normalization function for calculation;

[0030] The weight decay determines the loss function in the transformer architecture model training and the weight parameter of the low-rank adaptive matrix, adds a penalty term proportional to the square norm of the low-rank adaptive matrix weight to the initial loss function, generates the trained loss function, and guides the transformer architecture model training process.

[0031] Further, the training data set is generated by using the multi-level noise denoising neural network on the document corpus to be processed and the pre-configured knowledge base, and the heterogeneous graph is extracted from the training data set and the semantic entities in the document respectively to obtain the heterogeneous graph, including:

[0032] S31, based on the pre-configured domain knowledge base and the document corpus to be processed, the positive samples are marked, and the word level and sentence level denoising are processed to obtain the training data set;

[0033] S32, the training data set and the semantic entities in the document are extracted to obtain the heterogeneous graph of the training data set and the semantic entities in the document.

[0034] Further, the heterogeneous graph is fused by using the edge-oriented graph attention network model to obtain the relationship representation vector of the heterogeneous graph, and the relationship probability distribution of the relationship representation vector of the heterogeneous graph is performed by using the multilayer perception classifier to obtain the relationship result of the heterogeneous graph entity recognition, including:

[0035] S41, the edge-oriented graph attention network is used to process the heterogeneous graph;

[0036] S42, process the processed heterogeneous graph using a multi-level path attention mechanism, combine the importance of each path, and calculate the weight to obtain the relationship representation of the heterogeneous graph entity;

[0037] S43, input the relationship representation of the heterogeneous graph entity into a multi-layer perception classifier to obtain the probability distribution of each relationship of the entity, and obtain the relationship result of the heterogeneous graph entity recognition in the document.

[0038] Further, the co-reference resolution model is used to sequentially process the multi-modal document heterogeneous graph, semantic entity, heterogeneous graph, and relationship result of heterogeneous graph entity recognition, and combine the pre-trained link prediction model to calculate and threshold determine to obtain a global knowledge network, including:

[0039] S51, construct a global document graph taking the document as a node and the explicit reference, topic similarity, and shared term as an edge;

[0040] S52, process the global document graph through the co-reference resolution model, and combine the candidate triple verification to obtain a reliable global document graph;

[0041] S53, use the link prediction model to calculate the reliable global document graph to obtain a rationality score;

[0042] S54, based on the preset score threshold, the candidate triple higher than the preset threshold is considered reliable and automatically incorporated into the main knowledge graph; the triple lower than the preset threshold is marked as pending review and submitted for manual review or directly discarded.

[0043] Further, the standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding, and bounding box coordinate addition calculation formula are:

[0044] ;

[0045] In the formula, represents the final input embedding vector, represents the standard word embedding of the th text token, the one-dimensional position embedding of the th token, represents the two-dimensional layout position embedding of the th token, represents the logical role embedding of the th token, represents the bounding box coordinates of the th token on the document page, represents the position or index of the token in the input sequence, This refers to the text token itself. This represents the logical role of the text block to which the i-th token belongs. This represents the i-th text token.

[0046] Furthermore, the formula for calculating the value of each path by utilizing a multi-level path attention mechanism, processing the processed heterogeneous graph, and assigning weights based on the importance of each path is as follows:

[0047] ;

[0048] ;

[0049] In the formula, Indicates the first Attention weights for each path This represents an exponential function, where N represents the total number of paths connecting any two entity references in the document, and n represents the index variable in the denominator summation formula. This indicates the importance of calculating each path. Represents independent vector representations. Indicates all paths, This indicates the final relationship.

[0050] The beneficial effects of this invention are as follows:

[0051] 1) This invention analyzes document pages using an advanced visual language model (LayoutLMv3), jointly processing text, images, and layout information to identify basic layout elements such as text blocks, tables, and images. It innovatively assigns logical role labels to these elements, such as title, section heading, footnote, and list item. For complex elements like tables and mathematical formulas, specialized deep learning models are used for refined structure recognition. For example, a semantic segmentation model based on fully convolutional networks (FCNs) is used to parse the row and column structure of tables and the number of cells spanning rows and columns. A parser based on context-free grammars (CFGs) is used to convert formula images into structured representations such as MathML (Mathematical Markup Language). All parsed elements and attributes (content, position, logical role, internal structure) are integrated into a unified and rich multimodal document heterogeneous graph, providing input containing deep structural information for subsequent knowledge extraction tasks. Layout information, such as font, font size, and position, is used as model input to improve the accuracy of entity recognition.

[0052] 2) The application adopts a pre-trained visual language model (LayoutLMv3) as a basic architecture. For each text token, the input representation is the fusion of text semantic embedding, one-dimensional sequence position embedding, two-dimensional spatial coordinate embedding, and the aforementioned logical role embedding. The model can simultaneously perceive the meaning, order, position, and function of a word. Low-rank adaptation (Low-Rank Adaptation, LoRA) technology is used for efficient parameter fine-tuning. By injecting a small number of trainable low-rank matrices into specific layers of the model (such as the query, key, and value projection matrices of the self-attention module and the newly added layout-related embedding layer), the method achieves efficient adaptation to new domain standards while freezing most pre-trained parameters. It can reduce the number of trainable parameters by more than 90%, and effectively avoid overfitting when fine-tuning on small sample data. At the same time, attention dropout (Attention Dropout) and L2 weight decay regularization techniques are introduced during the fine-tuning process to further improve the stability and generalization ability of the model. The specific model for performing deep semantic entity extraction tasks. The model is a LayoutLMv3-base Transformer model (i.e. a Transformer architecture model) configured for the Token Classification task. The regularization technique is applied during the efficient parameter fine-tuning (LoRA) of the LayoutLMv3 model (i.e. a multi-modal document understanding model) to prevent the model from overfitting on a small amount of professional domain labeled data, thereby improving performance on new data. Automatically generate labeled data using domain rules and patterns to achieve relation extraction in the case of scarce labeled data.

[0053] 3) The application uses a framework to add a key multi-level noise reduction preprocessing step before model training, inspired by the idea of multi-level noise reduction neural network (MLNRNN). First, at the word level, an iterative keyword semantic aggregator (IKSA) is used to analyze each candidate relationship sentence, automatically identifying and reducing the weight of noise words unrelated to the target relationship. At the sentence level, all sentences containing the same entity pair are treated as a bag, and the multi-target multi-instance learning (MOMIL) algorithm is applied to identify the true positive sentence cluster that best represents the target relationship from the bag, and the remaining sentences are treated as false positives or noise samples. Through this word and sentence dual noise reduction mechanism, a significantly purer and more reliable training dataset can be provided for the subsequent relationship extraction model, thereby significantly improving the extraction accuracy of the model and accurately identifying and structuring complex structures such as tables, formulas, and lists.

[0054] 4) The invention is based on a document-based heterogeneous graph representation, containing nodes of entities, sentences, sections, table cells, etc. and various dependency edges between them, which serves as the input for relation extraction. Traditional graph neural networks (GNNs) focus on updating node representations differently, and the core of the EoGAT model (Edge-oriented Graph Attention Network model) is to learn edge representations. Through an iterative process, a unique path representation is generated for any two entities in the graph, which may have multiple paths between them, such as syntactic dependency paths, paths in the same table row, paths in the same section, etc. On this basis, the invention introduces a multi-level attention mechanism: a path-level attention layer is used to dynamically evaluate the importance of different paths for determining a specific relationship type and to perform weighted fusion. For example, for the performance indicator and test value relationship, the model may learn to assign higher weights to paths in the same table row. This allows the model to intelligently filter and integrate evidence from different levels, syntax, layout, and logical structure, effectively capturing complex relationships that traditional methods cannot discover, and enabling automated, end-to-end conversion from PDF documents to structured knowledge. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 is a flowchart of the deep semantic entity and relationship automatic extraction method for industry standard documents according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] To further illustrate the embodiments, the present application provides drawings, which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can explain the operating principle of the embodiments in conjunction with the related description of the specification. With reference to these contents, those skilled in the art should understand other possible embodiments and advantages of the present application.

[0058] According to an embodiment of the present application, a deep semantic entity and relationship automatic extraction method for industry standard documents is proposed.

[0059] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in the drawings, the deep semantic entity and relationship automatic extraction method for industry standard documents according to an embodiment of the present application comprises: Figure 1

[0060] ​Step S1, input the document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph;

[0061] Step S2, process the multi-modal document heterogeneous graph and the preset entity type through a transformer architecture model to extract embedding vectors of the multi-modal document heterogeneous graph and the preset entity type, and obtain semantic entities in the document;

[0062] The model for document parsing refers to a plurality of models, including LayoutLMv3 (i.e., a multi-modal document understanding model) for layout analysis, a fully convolutional network (FCN) model for table recognition, etc.

[0063] Step S3, generate a training data set using a multi-level noise reduction neural network on the document corpus to be processed and a pre-configured knowledge base, extract a heterogeneous graph from the training data set and the semantic entities in the document respectively, and obtain the heterogeneous graph;

[0064] The model for entity extraction refers to a LayoutLMv3-baseTransforme model (i.e., a transformer architecture model) for entity extraction.

[0065] Step S4, fuse the heterogeneous graph using an edge-oriented graph attention network model to obtain a relationship representation vector of the heterogeneous graph, and perform relationship probability distribution on the relationship representation vector of the heterogeneous graph through a multi-layer perception classifier to obtain a relationship result of heterogeneous graph entity recognition.

[0066] The model for relationship extraction refers to an edge-oriented graph attention network proposed by the present application.

[0067] Step S5, sequentially process the multi-modal document heterogeneous graph, the semantic entities, the heterogeneous graph, and the relationship result of the heterogeneous graph entity recognition using a coreference resolution model, and combine a pre-trained link prediction model to perform calculation and threshold determination to obtain a global knowledge network.

[0068] The model for knowledge proofreading refers to a link prediction (LinkPrediction) model, and the model refers to a LayoutLMv3 model for entity extraction. The general concept of the model in the embodiments is consistent, but the specific objects change with the tasks of the technical modules, and clear definitions are given in the embodiments above and below.

[0069] In this optional embodiment, the input of the document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph includes:

[0070] S11, input the document page image to be processed into a multi-modal document understanding model through integrated text and image processing capabilities to recognize all text blocks and visual elements on the page, and obtain a processed document page image;

[0071] S12, recognize the processed document page image through a fully convolutional network to obtain a document page chart, and combine structure recognition and content association to obtain structure data of the document page chart;

[0072] S13, use an attention mechanism encoder-decoder model and a context-free grammar parser to perform semantic parsing on the structure data to obtain a semantic representation of the structure data;

[0073] S14, output and integrate the processed document page image, the document page chart, the structure data of the document page chart, and the semantic representation of the structure data to construct a multi-modal document heterogeneous graph.

[0074] In this optional embodiment, the multi-modal document understanding model comprises:

[0075] extract text and function for classification, and pre-train using knowledge on a large number of documents, the multi-modal document understanding model predicts a logical role for each text block;

[0076] The logical roles include title, chapter title, footnote, header, footer, page number, and list item.

[0077] Specifically, a multi-modal document understanding model, specifically LayoutLMv3 in specific embodiments, is used as the core engine for document layout analysis. Unlike traditional methods that rely on optical character recognition (OCR), LayoutLMv3 can uniformly process information in three modalities: text, image, and layout.

[0078] When a document page image is input, the LayoutLMv3 model performs the following operations:

[0079] Content extraction: through integrated text and image processing capabilities, the model recognizes all text blocks and visual elements, such as pictures and dividing lines, on the page.

[0080] Geometric positioning: generate precise bounding box coordinates (x1, y1, x2, y2) for each recognized text block, where (x1, y1) is the top-left corner coordinate and (x2, y2) is the bottom-right corner coordinate.

[0081] Logical Role Classification: The model extracts text and classifies the function. Using knowledge learned from pre-training on a large corpus of documents, the model predicts a logical role for each text block. These roles include, but are not limited to: title, sectionHeading, footnote, pageHeader, pageFooter, pageNumber, and listItem.

[0082] The step of predicting a logical role is:

[0083] Extracting a candidate fact: When the system extracts a new knowledge triple from a document, the triple is not directly stored in the final knowledge graph, but is considered a candidate fact.

[0084] Calling a link prediction model: The candidate triple is input into a TransE link prediction model (i.e., a link prediction model) that has been pre-trained on a trusted knowledge graph.

[0085] Calculating a plausibility score: The link prediction model calculates a plausibility score for the candidate triple. The score reflects the compatibility or consistency of the triple with existing knowledge in the knowledge base. For example, if there are already a large number of triples in the knowledge base that are of the form less than device A, measure, performance indicator X greater than, then a newly extracted triple of the form less than device B, measure, performance indicator X greater than, where device B is similar in type to device A, will receive a higher plausibility score.

[0086] Decision and fusion: The calculated plausibility score is compared with a pre-set threshold.

[0087] If the score is higher than the threshold, the triple is considered to be trustworthy and will be automatically incorporated into the main knowledge graph.

[0088] If the score is lower than the threshold, the triple is marked as pending review and can be submitted to domain experts for manual review or discarded directly, thereby preventing the contamination of the knowledge base with false knowledge.

[0089] The specific steps are as follows:

[0090] Extracting a candidate fact: When the system extracts a new knowledge triple from a document, the triple is not directly stored in the final knowledge graph, but is considered a candidate fact.

[0091] Calling a link prediction model: The candidate triple is input into a TransE link prediction model (i.e., a link prediction model) that has been pre-trained on a trusted knowledge graph.

[0092] The above output is a series of structured page objects, each containing triple information of text content, geometric position and logical role. The introduction of logical role provides crucial clues for understanding the macro structure and semantic hierarchy of the document.

[0093] When a certain layout element in the above is identified as a table, the system passes the image region to a special Table Structure Recognition (TSR) submodule.

[0094] The specific implementation is as follows:

[0095] Model architecture: A Fully Convolutional Network (FCN) is used, which is trained to perform pixel-level classification on table images.

[0096] Segmentation target: The network outputs multiple channel segmentation masks, each corresponding to a class of structural elements, such as: row separator, column separator, header region.

[0097] Structure reconstruction: Post-processing of the segmentation masks output by the FCN. By analyzing the connected domains and relative positions of these masks, the complete grid structure of the table can be reconstructed. The algorithm can identify rows and columns, and accurately handle complex rowspan and columnspan cells.

[0098] Content association: The text content recognized by OCR is accurately filled into the reconstructed table grid according to the coordinates, and the header cells are associated with the corresponding data cells.

[0099] The table is converted into a graph-structured data containing row and column structure, cell content and header relationship, rather than a simple text block.

[0100] For the region in the document identified as a mathematical formula, a two-stage analysis process is used to convert it from image format to semantic structure representation.

[0101] Symbol sequence recognition: A encoder-decoder model based on attention mechanism (i.e. a convolutional neural network encoder plus a Transformer decoder) is used to input the image of the formula and output the corresponding symbol sequence, LaTeX string.

[0102] Syntax Parsing: The LaTeX string generated in the previous step is input into a context-free grammar (CFG) based parser. The parser constructs a linear sequence of symbols into an Abstract Syntax Tree (AST) based on predefined mathematical grammar rules. This tree explicitly defines the operation structure of the formula, operators, operands, subscript relationships, etc. The final output is a standardized LaTeX representation that can be understood by machines to understand the mathematical meaning.

[0103] The above processing procedures are collectively referred to as document table structure and semantic parsing processing.

[0104] This module integrates the outputs of all the above steps to build a global, multi-modal document heterogeneous graph = (V, E).

[0105] Node Set (V): The types of nodes are heterogeneous, including: Paragraph (paragraph), Heading (title), Table (table), TableCell (table cell), Formula (formula), Figure (picture), etc. Each node stores corresponding attributes such as text content, bounding box, logical role, etc.

[0106] Edge Set (E): The types of edges are also diverse, representing the relationships between different elements:

[0107] ReadingOrder: Connects consecutive text blocks to represent the reading order of the document.

[0108] HierarchicalContainment: Represents the logical containment relationship, for example, the TableCell node points to the Table node it belongs to, and the Paragraph node points to the Heading node it belongs to.

[0109] ExplicitReference: Represents explicit reference relationships, for example, the Paragraph node points to the corresponding Table node.

[0110] Heterogeneous Graph is the unified input of all subsequent knowledge extraction modules of the invention, retaining the rich structure and semantic information of the document in a way far beyond pure text.

[0111] In this optional embodiment, the multi-modal document heterogeneous graph and the preset entity type are processed through the transformer architecture model to extract the embedding vectors of the multi-modal document heterogeneous graph and the preset entity type, and the semantic entities in the document are obtained.

[0112] S21, taking the multi-modal document heterogeneous graph and the preset entity type as core input data, extracting embedding vectors of the multi-modal document heterogeneous graph and the preset entity type through standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding and boundary box coordinate addition calculation;

[0113] S22, processing the extracted embedding vectors through a low-rank adapter and a regularization technique in sequence to obtain semantic entities in the document.

[0114] In this optional embodiment, the regularization technique comprises:

[0115] attention dropout and weight decay;

[0116] Among them, the attention dropout inputs the extracted embedding vectors into attention calculation to obtain attention scores, selects part of the attention scores to be zero through random selection, and inputs the attention scores into a normalization function for calculation;

[0117] The weight decay determines the loss function in the transformer architecture model training and the weight parameters of the low-rank adaptive matrix, adds a penalty term proportional to the square norm of the low-rank adaptive matrix weight to the initial loss function, generates the loss function after training, and guides the transformer architecture model training process.

[0118] Specifically, in order to solve the problem of fine-tuning on a small amount of labeled data, low-rank adaptive (LoRA) fine-tuning: through LoRA technology, during training, the original most weights of the pre-trained model remain unchanged, and only small, trainable "adapter" modules are injected into specific layers of the Transformer architecture. Specifically, for a weight matrix , the update amount is decomposed into the product of two low-rank matrices , where and , and the rank is much smaller than the original dimensions and During fine-tuning, only the parameters of matrices A and B are trained. is decomposed into the product of two smaller (low-rank) matrices.

[0119] and is a standard mathematical symbol representing the set of real numbers. Each element in matrix A is a real number. In the field of machine learning and deep learning, it is a very common representation method because the weights of the model (such as matrix A) are usually represented by real numbers.

[0120] is the original weight matrix in the pre-trained model that is frozen during the fine-tuning process.

[0121] This represents the amount of weight updates that need to be learned during fine-tuning. The final model weights are... .

[0122] B and A constitute the update quantity. Two low-rank matrices, namely During fine-tuning, only the parameters of these two small matrices A and B are trained, instead of the entire huge matrix. matrix.

[0123] The rank is the rank of the low-rank decomposition. A hyperparameter much smaller than the dimension of the original matrix (in the embodiments of this invention, ...). ). The value of determines the number of trainable parameters and is key to achieving efficient parameter fine-tuning. In this embodiment, MLNRNN is an abbreviation for Multi-level Noise Reduction Neural Network. This illustrates how multi-level noise reduction preprocessing in the weakly supervised relation extraction framework identifies and suppresses noise at different granularities, such as word and sentence levels.

[0124] The selection of LoRA target modules depends on the application location of the LoRA adapter (i.e., the low-rank adapter). To maximize the synergy between text and layout information, LoRA is strategically applied to the following modules:

[0125] The self-attention mechanism includes query, key, and value projection layers. A two-dimensional position embedding layer is added to integrate layout information. ) and logical role embedding layer ( This design forces the model's adaptation to focus on learning how to adjust attention patterns based on text and layout / character cues, achieving a deep and efficient fusion of the two modalities.

[0126] Regularization techniques: To prevent overfitting and improve model robustness during LoRA fine-tuning, this invention integrates two regularization methods:

[0127] Attention Dropout: After calculating the attention weights and before applying the softmax function, a portion of the attention scores are randomly set to zero. This prevents the model from becoming overly reliant on a few attention heads and promotes greater diversity in attention patterns.

[0128] L2 Weight Decay: Add a penalty term in the loss function, which is proportional to the square of the L2 norm of the weights of LoRA matrices (A and B). It can effectively suppress the weight values from becoming too large, thereby improving the generalization ability of the model. As shown in Table 1:

[0129] Table 1 Implementation parameters

[0130]

[0131] In this optional embodiment, the multi-level noise reduction neural network is used to generate a training data set from the pre-configured knowledge base and the document corpus to be processed, and the semantic entities in the training data set and the document are extracted into heterogeneous graphs, and the heterogeneous graphs include:

[0132] S31, based on the pre-configured domain knowledge base and the document corpus to be processed, mark as positive samples, and process through word-level and sentence-level noise reduction, to obtain a training data set;

[0133] S32, extract the heterogeneous graph of the semantic entity in the training data set and the document, to obtain the heterogeneous graph of the semantic entity in the training data set and the document.

[0134] Specifically, two core problems of relationship extraction are solved, how to reduce the dependence on manual annotation, and how to accurately extract complex relationships across long text distances. The multi-level noise reduction training data generation of remote supervision:

[0135] In order to automatically generate training data for relationship extraction, the remote supervision method is adopted, but the key noise suppression mechanism is introduced to solve the inherent defects of the hypothesis.

[0136] Initial data annotation, i.e. remote supervision: Use an existing domain knowledge base or ontology, for example, a database containing standard part parameters and relationships. For each relationship triple in the knowledge base, entity1, relationship, entity2, search all sentences containing entity1 and entity2 in the document corpus to be processed. The sentence is preliminarily and automatically marked as a positive sample expressing the relationship.

[0137] Multi-level noise reduction: Before using the automatically annotated data for model training, the multi-level noise reduction neural network performs multi-level noise reduction preprocessing steps in the weakly supervised relationship extraction framework to identify and suppress noise at different granularities, such as word-level and sentence-level.

[0138] Word-level denoising: For each sentence that is initially labeled, an Iterative Keyword Semantic Aggregator (IKSA) is applied. IKSA computes an importance weight for each word in the sentence by analyzing the semantic relevance of the word to the target entity pair and the relation. Context words that are not relevant to the relation expression, such as "in addition", "however", etc. are given lower weights. This is used to filter out irrelevant information within a sentence and highlight the key words that actually carry the relation information.

[0139] Sentence-level denoising: All sentences that contain the same entity pair are grouped into a bag. A Multi-Objective Multi-Instance Learning (MOMIL) algorithm is applied to the bag to filter out sentences that are not relevant to the relation expression. MOMIL aims to identify one or more true positive sentences that clearly and unambiguously express the target relation within the bag. This is achieved by clustering or similarity computation in the feature space, which identifies sentences that contain the entity pair but do not actually express the relation (i.e. noise) and removes or down-weights them from the training set.

[0140] Through the two-step denoising process of pollution first and then governance, a much higher quality training dataset is generated than traditional distant supervision methods, laying a solid foundation for training high-precision relation extraction models.

[0141] Heterogeneous graph construction for relation extraction: For each document, a heterogeneous graph dedicated to relation extraction is constructed = (V, E), the structure of the heterogeneous graph is more detailed than the document graph in Module 1, focusing on the connection between entities.

[0142] Nodes (V):

[0143] Mention: represents a specific mention of an entity in the document.

[0144] Entity: represents a node of an abstract entity, and all Mention nodes of the same entity point to E.

[0145] Sentence: represents a sentence, and the Mention node is linked to the Sentence node where it is located.

[0146] Section: represents a chapter or section, and the Sentence node is linked to the Section node to which it belongs.

[0147] TableCell: if the entity mention appears in a table, the Mention node is linked to the corresponding TableCell node.

[0148] Edges (E):

[0149] Containment: Connects mentions of various containment relations, e.g., mentions of sentences, chapters, etc.

[0150] Co-occurrence: Connects mentions that occur in the same sentence / chapter / table row.

[0151] Syntactic Dependency: Connects pairs of mentions that have a direct syntactic relationship based on the syntactic dependency parse tree.

[0152] Coreference: Connects different mentions of the same entity 0, e.g., device and spectrum analyzer.

[0153] Edge-oriented graph attention network with multi-level attention: To realize relation extraction across sentences or even paragraphs, the edge-oriented graph attention network.

[0154] Edge-oriented paradigm: Existing graph attention networks (GAT) update the representation of the center node by aggregating the information of neighboring nodes. The relationship between entities is more reflected in the connection path. Adopting the edge-oriented (edge-oriented) paradigm, the core computing unit of the model is to update the representation of the edge, not the node. The representation of an edge will be iteratively updated according to the information of the two connected nodes and the adjacent edges.

[0155] Multi-path representation learning: In a heterogeneous graph , there may be multiple paths of different types between any two entity mentions and . For example:

[0156] Syntactic path: greater than subject-predicate relationship greater than v greater than object-verb relationship greater than .

[0157] Table path: greater than greater than same row greater than greater than .

[0158] Chapter co-occurrence path: greater than greater than greater than greater than . The EoGAT model learns an independent vector representation for each such path , which encodes the structural and semantic information of the path.

[0159] Multi-level path attention mechanism: This is the core mechanism designed by the invention to solve complex relation extraction. After obtaining an entity pair ( , All paths between .

[0160] IKSA (Iterative Keyword Semantic Aggregator) is an algorithm module used for word-level noise reduction before weakly supervised relation extraction.

[0161] Meaning: For a sentence that may contain a target relation, the algorithm automatically evaluates the relevance of each word in the sentence to the target relation and assigns different importance weights. The purpose is to filter out irrelevant information within the sentence and highlight the key words that truly carry relationship information.

[0162] and : These two symbols are used in this embodiment to illustrate the sentence nodes (SentenceNode) when constructing paths. For example, in the example of chapter co-occurrence paths, a path may contain the nodes where entities are located , forming intermediate nodes connecting the two entity paths.

[0163] represents the table cell node (TableCellNode) when constructing paths. For example, in a table path connected by table structure,

[0164] represents the table cell node where the entity is located.

[0165] The model can adaptively learn the weights of different types of evidence paths according to the type of relationship to be predicted. For example, when predicting dependency relations, the model will learn to give higher attention weights to syntactic paths; while predicting parameters belonging to the same group, it will pay more attention to table paths.

[0166] Relation classification: The final relationship representation is fed into a multi-layer perceptron (MLP) classifier, which outputs the probability distribution of various possible relationships between entity pairs through a Softmax layer.

[0167] Specific analysis steps: For a candidate relationship sentence, for example, a sentence containing both the entity insulation resistance and the test voltage, to determine whether it expresses the test condition, the analysis steps of IKSA are as follows:

[0168] Input: a candidate sentence, the identified entity pair in the sentence, for example, insulation resistance, test voltage, and a target relationship, for example, test condition.

[0169] Semantic relevance analysis: IKSA analyzes the relevance by computing the semantic similarity between the vector representation of each word in the sentence and the vector representation of the target entity pair and the target relation.

[0170] Weight computation: Based on the semantic relevance score, a weight of importance is computed for each word in the sentence. Words that are closely related to the expression of the target relation, e.g. under, will get a high weight, while context words that are irrelevant, e.g. moreover, however, as shown in the figure, etc. will get a lower weight.

[0171] Iterative optimization: The above process can be repeated, using the high-weighted words computed in the last round to further focus and optimize the weights of other words, until the weight distribution converges to a stable state.

[0172] Output: A sentence representation with word weight information. The weighted sentence representation will be used for the training of subsequent sentence-level denoising or relation extraction models. Since the weights of noisy words have been reduced, the training of the model will be more robust.

[0173] In the optional embodiment, the edge-oriented graph attention network model is used to fuse the heterogeneous graphs to obtain a relationship representation vector of the heterogeneous graphs, and a multi-layer perception classifier is used to obtain a relationship result of entity recognition of the heterogeneous graphs by performing relationship probability distribution on the relationship representation vector of the heterogeneous graphs.

[0174] S41, processing the heterogeneous graphs by using an edge-oriented graph attention network;

[0175] S42, processing the processed heterogeneous graphs by using a multi-level path attention mechanism, combining the importance of each path, and calculating and fusing to obtain a relationship representation of the entities of the heterogeneous graphs;

[0176] S43, inputting the relationship representation of the entities of the heterogeneous graphs into a multi-layer perception classifier to obtain a probability distribution of each type of relationship of the entity pairs, and obtaining a relationship result of entity recognition of the heterogeneous graphs in the document.

[0177] Specifically, the knowledge extracted from a single document is integrated to construct a consistent and accurate knowledge network covering the entire standard system.

[0178] The knowledge in the industry standard system is often scattered in multiple mutually referenced documents. In order to link the above isolated knowledge, an advanced cross-document coreference resolution (CDCR) technology is adopted.

[0179] Global document graph construction: construct a global document graph with documents as nodes and explicit references, topic similarity, or shared terms as edges.

[0180] In this optional embodiment, the co-reference resolution model is used to process the multi-modal document heterogeneous graph, semantic entity, heterogeneous graph, and relationship result of heterogeneous graph entity recognition in turn, and the pre-trained link prediction model is used for calculation and threshold determination to obtain a global knowledge network, including:

[0181] S51, a global document graph is constructed with documents as nodes and explicit references, topic similarity, and shared terms as edges.

[0182] S52, the global document graph is processed by a co-reference resolution model, and combined with candidate triple verification to obtain a reliable global document graph.

[0183] S53, the link prediction model is used to calculate the reliable global document graph to obtain a plausibility score.

[0184] S54, based on the preset score threshold, the candidate triple higher than the preset threshold is considered reliable and automatically incorporated into the main knowledge graph; the triple lower than the preset threshold is marked as pending review and submitted for manual review or directly discarded.

[0185] Specifically, based on the probabilistic knowledge correction of link prediction, in order to ensure the quality of the final knowledge in the database and prevent extraction errors from polluting the knowledge graph.

[0186] Candidate triple verification: when the system extracts a new knowledge triple from the document, less than entity 1, relationship, entity 2 greater than, the triple is not directly stored in the database and is considered as a candidate fact.

[0187] Link prediction model: the model can be based on a classic graph embedding model or a more powerful graph neural network-based model, such as TransE, ComplEx (i.e. knowledge graph embedding).

[0188] Plausibility score: the link prediction model calculates a plausibility score for the candidate triple, which reflects the compatibility of the triple with the existing knowledge in the knowledge base. For example, if there are a large number of facts less than device A, measurement, performance indicator X greater than in the knowledge base, then the newly extracted less than device B, measurement, performance indicator X greater than, where device B is similar to device A in type, will obtain a higher score.

[0189] Decision and fusion: Set a reasonable score threshold. Candidate triples above the threshold are considered reliable and automatically incorporated into the main knowledge graph. Triples below the threshold are marked for review and submitted to domain experts for manual review or discarded directly.

[0190] EoGAT model is used to solve the problem of complex relationship extraction across sentences and paragraphs.

[0191] Specific reference and inclusive content: EoGAT model is an edge-oriented graph attention network. EoGAT model is a novel graph neural network (GNN) architecture. Unlike traditional GNNs, which focus on updating the representation of graph nodes such as entities, the core paradigm of EoGAT is edge-oriented, that is, the core computing unit is to learn and update the representation of edges in the graph, which represents the relationship path between entities.

[0192] Mainly contains three core mechanisms:

[0193] Edge-oriented update paradigm: The focus of the model's calculation is to update the representation of edges, not nodes.

[0194] Multi-path representation learning: It can learn independent vector representations for multiple different types of paths connecting two entities (such as syntactic paths, table paths, etc.).

[0195] Multi-level path attention mechanism: Through an attention layer, dynamically evaluate and fuse the importance of different paths for judging a specific relationship type.

[0196] Specific steps based on EoGAT model:

[0197] Input construction: Based on the document, build a heterogeneous graph dedicated to relationship extraction The heterogeneous graph contains multiple types of nodes such as entity mentions, sentences, chapters, table cells, and multiple dependency relationship edges between them.

[0198] Multi-path identification and representation learning: For any two entity mentions and , the EoGAT model will identify K different paths connecting them, such as syntactic dependency paths, same table row paths, and same chapter co-occurrence paths. The model will learn an independent vector representation hk for each path k, which encodes the structural and semantic information of the path.

[0199] Multi-level attention for path importance evaluation: A path-level attention module will calculate the importance of each path representation Importance of final relation prediction. This is done by a learnable function that computes attention weights for each path The weights are dynamic, e.g., the model can learn to assign higher weights to paths that are more consistent with the table row when predicting performance metrics and test values relations.

[0200] Path information fusion: vector representations of all paths are fused According to the corresponding attention weights A comprehensive representation of the entity pair is obtained by weighted summing the final representation vectors of the relations between them .

[0201] Relation classification: the final representation vector of the entity pair is fed into a standard multi-layer perceptron (MLP) classifier, and the probability distribution of all predefined relations between the entity pair is output through a Softmax layer, thus completing the relation prediction. Prediction refers to the step of using a Link Prediction model to evaluate and correct the newly extracted knowledge triples in the cross-document knowledge fusion and correction.

[0202] The specific steps are as follows:

[0203] Obtain candidate facts: extract a new knowledge triple from the document, e.g., Entity1, Relation, Entity2, which is not directly stored in the final knowledge graph, but is considered as a candidate fact.

[0204] Call the link prediction model: input the candidate triple into a TransE link (link prediction model) prediction model pre-trained on a trusted knowledge graph.

[0205] Calculate the plausibility score: the link prediction model calculates a plausibility score for the candidate triple. The score reflects the compatibility or consistency of the triple with the existing knowledge in the knowledge base. For example, if there are a large number of facts in the knowledge base that Entity1, Measure, Performance Metric X are greater than Entity2, then a new extracted triple that Entity1, Measure, Performance Metric X is greater than Entity2, which is similar in type to Entity1, will obtain a higher plausibility score.

[0206] Decision and fusion: compare the calculated plausibility score with a pre-set threshold.

[0207] If the score is higher than the threshold, the triple is considered to be trustworthy and will be automatically incorporated into the main knowledge graph.

[0208]

[0209] ​If the score is below the threshold, the triplet is marked as pending review and can be submitted to scholars for manual review or discarded directly, thereby preventing erroneous knowledge from polluting the knowledge base.

[0210] In this optional embodiment, the model is a LayoutLMv3-baseTransformer model (i.e., a transformer architecture model) configured for the token classification task. Its innovation lies in the unique input embedding layer design, which deeply integrates text and layout information. For the i-th token in the input sequence... The final input embedding vector It is composed of the following four parts. The formula for calculating the sum of the standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding, and bounding box coordinates is as follows:

[0211] ;

[0212] In the formula, This represents the final input embedding vector. Indicates the first The standard word embedding of a text token, No. One-dimensional position embedding of each token Indicates the first The two-dimensional layout of each token is embedded in the position. Indicates the first The logical role embedding of each token, Indicates the first The bounding box coordinates of the tokens on the document page. Indicates the position or index of the token in the input sequence. This refers to the text token itself. This represents the logical role of the text block to which the i-th token belongs. This represents the i-th text token.

[0213] The final input embedding vector represents the semantic information of the token itself, obtained by querying the model's vocabulary.

[0214] Indicates the first The standard word embeddings of a text token are obtained by querying the vocabulary of the model and represent the semantic information of the token.

[0215] Indicates the first The one-dimensional positional embedding of each token represents the order information of the tokens within the entire text sequence.

[0216] Indicates the first The two-dimensional layout of each token is embedded, representing its physical spatial location on the document page, based on the token's bounding box coordinates (bbox). i It was calculated.

[0217] This indicates the position or index of a token in the input sequence, described in the documentation as the i-th token.

[0218] and This refers to the text token itself. In the formula, it refers to the ordinal word embedding of the i-th text token. This represents the i-th text token.

[0219] This represents the logical role of the text block to which the i-th token belongs. For example, this role could be a title, list item, or footnote.

[0220] Token bounding box coordinates Normalized to the integer range. Each coordinate component (left, top, width, height) is mapped to a vector through an independent trainable embedding layer. The vectors are concatenated or added to form an embedding vector representing the token's absolute position on the page.

[0221] Indicates the first The logical role embedding of each token represents the functional role of the text block to which the token belongs in the document structure, such as title, list item, footnote, etc. The role information is obtained by parsing from Module 1.

[0222] The logical role embedding represents the logical role of the text block containing the token identified by Module 1, such as sectionHeading (i.e., chapter title), which is mapped to a trainable embedding vector. Embeddings provide the model with powerful contextual cues; for example, the model can learn that text appearing in the sectionHeading role is more likely to be an important technical requirement entity.

[0223] In this way, when processing each token, the model can simultaneously obtain information about what it is (semantics), where it is (order), where it is on the page (space), and what type of content block it is in (function), thereby greatly enhancing the accuracy of entity recognition.

[0224] Indicates the first The bounding box coordinates of a token on a document page are typically represented as a set of coordinate values, such as x, y for the top left corner and x, y for the bottom right corner, which precisely define the position and range of the token on the two-dimensional page.

[0225] In this optional embodiment, a path-level attention module is used to compute the final relation representation. The formula for calculating the heterogeneous graph by utilizing a multi-level path attention mechanism and processing the graph, and combining the importance of each path with weights, is as follows:

[0226] ;

[0227] ;

[0228] In the formula, Indicates the first Attention weights for each path This represents an exponential function, where N represents the total number of paths connecting any two entity references in the document, and n represents the index variable in the denominator summation formula. This indicates the importance of calculating each path. Represents independent vector representations. Indicates all paths, This indicates the final relationship.

[0229] Specifically, The exponential function is the core component of the Softmax function in this formula. It maps the score of each path to a positive number and widens the gap between different scores, allowing the model to focus more clearly on the most important paths.

[0230] N represents the total number of paths connecting any two entity mentions in two entity mention documents. and The total number of paths between entities. Identify all possible paths connecting two entities in the document's heterogeneity graph, such as syntactic paths, table paths, chapter co-occurrence paths, etc. It is the sum of these paths.

[0231] n represents the index variable in the summation formula of the denominator, traversing from 1 to N, representing each specific path.

[0232] It represents a small feedforward network used to calculate the importance of each path.

[0233] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatic extraction of deep semantic entities and relations for industry standard documents, characterized in that, The method comprises the following steps: S1, inputting a document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph; comprising: S11, inputting the document page image to be processed into the multi-modal document understanding model through integrated text and image processing capabilities to identify all text blocks and visual elements on the page to obtain a processed document page image; S12, identifying the processed document page image through a full convolutional network to obtain a document page chart, and combining structure recognition and content association to obtain structure data of the document page graph; S13, using an attention mechanism encoder-decoder model and a context-free grammar parser to perform semantic parsing on the structure data to obtain a semantic representation of the structure data; S14, outputting and integrating the processed document page image, the document page chart, the structure data of the document page graph, and the semantic representation of the structure data to construct a multi-modal document heterogeneous graph; S2, processing the multi-modal document heterogeneous graph and a preset entity type through a transformer architecture model to extract embedding vectors of the multi-modal document heterogeneous graph and the preset entity type to obtain semantic entities in the document; comprising: S21, taking the multi-modal document heterogeneous graph and the preset entity type as core input data, and extracting embedding vectors of the multi-modal document heterogeneous graph and the preset entity type through standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding, and boundary box coordinate addition calculation; S22, processing the extracted embedding vectors through a low-rank adapter and a regularization technique in sequence to obtain semantic entities in the document; S3, using a multi-level noise reduction neural network to generate a training data set from a document corpus to be processed and a pre-configured knowledge base, and extracting a heterogeneous graph from the training data set and the semantic entities in the document respectively to obtain a heterogeneous graph; comprising: S31, marking the pre-configured domain knowledge base and the document corpus to be processed as positive samples, and processing through word-level and sentence-level noise reduction to obtain a training data set; S32, extracting a heterogeneous graph from the training data set and the semantic entities in the document to obtain a heterogeneous graph of the training data set and the semantic entities in the document; S4, using an edge-oriented graph attention network model to fuse the heterogeneous graph to obtain a relationship representation vector of the heterogeneous graph, and performing relationship probability distribution on the relationship representation vector of the heterogeneous graph through a multilayer perceptron classifier to obtain a relationship result of entity recognition of the heterogeneous graph; comprising: S41, processing the heterogeneous graph using an edge-oriented graph attention network; S42, processing the processed heterogeneous graph using a multi-level path attention mechanism, combining the importance of each path, and calculating the weight to obtain a relationship representation of the entities of the heterogeneous graph; S43, inputting the relationship representation of the entities of the heterogeneous graph into a multilayer perceptron classifier to obtain a probability distribution of each type of relationship of the entities, and obtaining a relationship result of entity recognition of the heterogeneous graph in the document; S5, utilize the co-reference resolution model to process the multi-modal document heterogeneous graph, semantic entity, heterogeneous graph and the relationship result of heterogeneous graph entity recognition in turn, and combine the pre-trained link prediction model to calculate and threshold determination, obtain the global knowledge network.

2. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The multi-modal document understanding model comprises: Extract text and function for classification, and pre-train using knowledge on a large number of documents, the multi-modal document understanding model predicts a logical role for each text block; The logical role includes title, chapter title, footnote, header, footer, page number and list item.

3. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The regularization technique comprises: Attention dropout and weight decay; Among them, the attention dropout, the attention score obtained by inputting the extracted embedding vector into the attention calculation, part of the attention score is zero by random selection, and the attention score is input into the normalization function for calculation; Weight decay, determine the loss function and the weight parameter of the low rank adaptive matrix in the transformer architecture model training, add a penalty term proportional to the square norm of the weight of the low rank adaptive matrix to the initial loss function, generate the loss function after training, and guide the transformer architecture model training process.

4. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The utilization of co-reference resolution model to process the multi-modal document heterogeneous graph, semantic entity, heterogeneous graph and the relationship result of heterogeneous graph entity recognition in turn, and combine the pre-trained link prediction model to calculate and threshold determination, obtain the global knowledge network comprises: S51, construct a global document graph taking document as node and explicit reference, theme similarity and shared term as edge; S52, process the global document graph through the co-reference resolution model, and combine the candidate triple verification to obtain a reliable global document graph; S53, calculate the reasonable score of the reliable global document graph by using the link prediction model; S54, based on the preset score threshold, the candidate triple higher than the preset threshold is considered as reliable and automatically incorporated into the main knowledge graph; The triple lower than the preset threshold is marked as to be reviewed and submitted for manual review or directly discarded.

5. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding and boundary box coordinate addition calculation formula are: ； wherein, denotes the final input embedding vector, denotes the standard word embedding of the th text token, denotes the one-dimensional position embedding of the th token, denotes the two-dimensional layout position embedding of the th token, denotes the logical role embedding of the th token, denotes the bounding box coordinates of the th token on the document page, denotes the position or index of the token in the input sequence, denotes the text token itself, denotes the logical role of the i th token belonging to a text block, denotes the i th text token.

6. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The formula for processing the processed heterogeneous graph by using the multi-level path attention mechanism and combining the importance of each path and assigning weights to calculate is: ； ； wherein, denotes the attention weight of the th path, denotes the exponential function, N denotes the total number of paths connecting any two entity mentions in a document, n denotes the index variable in the denominator summation formula, denotes the importance of each path, denotes the independent vector representation, denotes all paths, denotes the final resulting relation.

Images