Industrial standard document-oriented deep semantic entity and relation automatic extraction method

Abstract

The invention discloses an industry standard document-oriented deep semantic entity and relationship automatic extraction method, which relates to the technical field of document intelligent processing, and comprises the following steps of: inputting a document page image into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph; processing the converter architecture model to obtain a semantic entity; generating a training data set by the multistage noise reduction neural network, and extracting a heterogeneous graph to obtain the heterogeneous graph; fusing the edge-oriented graph attention network model to obtain a relation result of heterogeneous graph entity recognition; and the co-reference resolution model processes a relation result identified by the heterogeneous graph, the semantic entity, the heterogeneous graph and the heterogeneous graph entity of the multi-modal document, and a global knowledge network is obtained by combining calculation of the trained link prediction model. According to the method, a purer and more reliable training data set is provided, and the extraction precision of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent document processing, in particular to a deep semantic entity and relationship automatic extraction method for industry standard documents. BACKGROUND

[0002] Industry standard documents are important technical documents that guide industrial production, guarantee product quality, and promote industrial development. The documents are usually presented in PDF format and contain a large amount of structured and semi-structured knowledge, such as technical requirements, performance indicators, test methods, and equipment parameters. Efficient and accurate extraction of these knowledge from the documents and construction of a machine-readable knowledge graph have great value for intelligent question answering, compliance checking, technical research and development, and knowledge management, and are located in the fields of artificial intelligence, natural language processing, knowledge graph construction, and document intelligence processing. However, the existing technology faces the following severe challenges when processing such documents: difficulty in cross-document knowledge association, large industry standard system, and a large number of reference and relationship between different standards. The existing methods are difficult to automatically identify these cross-document references and relationships, resulting in isolated knowledge and inability to construct a complete industry knowledge network.

[0003] Low efficiency and poor generalization in domain adaptation. Although pre-training language models represented by BERT have achieved remarkable results in natural language processing tasks, when applied to highly specialized industry standard fields, complete model fine-tuning is usually required, and a large amount of domain annotation data, high computing cost and storage overhead are required. For each new industry or new standard system, a repeated high-cost process is required, which limits the scalability and economy of the technical solution. At the same time, in the case of scarce annotation data, full fine-tuning can easily cause the model to overfit in a specific domain, losing the ability to generalize.

[0004] In addition, existing entity and relationship extraction methods rely heavily on surface text features and are difficult to understand the deep semantics in standard documents, such as the inability to distinguish between different types of parameter entities and the inability to identify the implicit relationship between test methods and test results. The noise problem in weakly supervised relationship extraction is prominent. To reduce the dependence on manual annotation, distant supervision (Distant Supervision) or weak supervision methods are widely used in relationship extraction. The core assumption is that if there is a relationship r between entity pair (e1, e2) in the knowledge base, all sentences containing both entity mentions express relationship r. However, this assumption is too strict in practice and will introduce a lot of noise labels. For example, a sentence may mention both the insulation resistance and the test current, but describe two different parameters respectively, and does not express the test condition relationship between them. Traditional methods require a large amount of manual annotation to train the model, and the standard document field is strong, which not only has a high annotation cost, but also requires annotators to have deep domain knowledge, resulting in low annotation efficiency.

[0005] The modeling capability for long-distance and implicit relationships is insufficient, and the entity relationship in the industry standard often spans multiple sentences, paragraphs, or even chapters. For example, the technical parameters defined in the first chapter may be detailed in a table in the fifth chapter. Existing relationship extraction models, such as end-to-end models based on LSTM or Transformer (i.e., deep learning models in the field of artificial intelligence), can handle intra-sentence relationships, but have limited capability in modeling complex dependency relationships of long-distance and cross-structure. Multiple models usually treat documents as flat sequences or simple graph structures, making it difficult to capture deep semantic associations determined by document logical levels such as chapter-paragraph-list and spatial layout and table rows and columns. Traditional methods often treat PDF documents as linear text streams, ignoring rich visual formatting information. There are a large number of tables, nested lists, charts, mathematical formulas, and column layouts in industry standard documents. Simple text extraction techniques cannot handle these complex two-dimensional structures, which can easily lead to loss of key information or structural disorder, such as failure to correctly associate table headers with cell contents or failure to identify the hierarchical relationship of nested lists, making subsequent information extraction tasks incomplete from the source, and unable to achieve unified and structured storage of information.

[0006] The existing technology has a series of problems such as shallow layout understanding, high domain adaptation cost, large noise interference of weak supervision, and weak complex relationship modeling capability when dealing with industry standards and other complex documents, making it difficult to achieve automated, high-precision, and efficient knowledge extraction.

[0007] Currently, there is no effective solution to the problems in the related art. SUMMARY

[0008] To solve the problems in the related art, the present application proposes a deep semantic entity and relationship automatic extraction method for industry standard documents to overcome the above technical problems existing in the prior art.

[0009] To this end, the specific technical solutions adopted by the present application are as follows: The present application proposes a deep semantic entity and relationship automatic extraction method for industry standard documents, comprising: S1, inputting a document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph; S2, processing the multi-modal document heterogeneous graph and the preset entity type through a transformer architecture model to extract embedding vectors of the multi-modal document heterogeneous graph and the preset entity type, and obtaining semantic entities in the document; S3, generating a training data set by using a multi-level noise reduction noise neural network on the document corpus to be processed and a pre-configured knowledge base, extracting an heterogeneous graph for the semantic entity in the training data set and the document respectively, and obtaining the heterogeneous graph; S4, fusing the heterogeneous graph by using an edge-oriented graph attention network model, obtaining a relationship representation vector of the heterogeneous graph, and performing relationship probability distribution on the relationship representation vector of the heterogeneous graph by using a multi-layer perception classifier, and obtaining a relationship result of the heterogeneous graph entity recognition; S5, processing the multi-modal document heterogeneous graph, the semantic entity, the heterogeneous graph and the relationship result of the heterogeneous graph entity recognition in sequence by using a co-reference resolution model, and performing calculation and threshold determination by combining a pre-trained link prediction model, and obtaining a global knowledge network.

[0010] Further, the processing of the document page image to be processed into the multi-modal document understanding model to obtain the multi-modal document heterogeneous graph comprises: S11, inputting the document page image to be processed into the multi-modal document understanding model by integrating text and image processing capabilities to recognize all text blocks and visual elements on the page, and obtaining a processed document page image; S12, recognizing the processed document page image by using a full convolutional network to obtain a document page chart, and combining structure recognition and content association to obtain structure data of the document page graph; S13, performing semantic parsing on the structure data by using an attention mechanism encoder-decoder model and a context-free grammar parser to obtain a semantic representation of the structure data; S14, outputting and integrating the processed document page image, the document page chart, the structure data of the document page graph and the semantic representation of the structure data to construct a multi-modal document heterogeneous graph.

[0011] Further, the multi-modal document understanding model comprises: extracting text and function for classification, and pre-training on a large amount of document knowledge, the multi-modal document understanding model predicting a logical role for each text block; The logical roles include title, chapter title, footnote, header, footer, page number and list item.

[0012] Further, the processing of the multi-modal document heterogeneous graph and the preset entity type by using the transformer architecture model to extract the embedding vector of the multi-modal document heterogeneous graph and the preset entity type to obtain the semantic entity in the document comprises: S21, taking the multi-modal document heterogeneous graph and the preset entity type as core input data, extracting embedding vectors of the multi-modal document heterogeneous graph and the preset entity type through standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding and boundary box coordinate addition calculation; S22, processing the extracted embedding vectors through a low-rank adapter and a regularization technique in sequence to obtain semantic entities in the document.

[0013] Further, the regularization technique comprises: attention dropout and weight decay; Among them, the attention dropout inputs the extracted embedding vectors into the attention calculation to obtain attention scores, selects part of the attention scores to be zero by random selection, and inputs the attention scores into a normalization function for calculation. The weight decay determines the loss function in the transformer architecture model training and the weight parameters of the low-rank adaptive matrix, adds a penalty term proportional to the square norm of the low-rank adaptive matrix weight to the initial loss function, generates the trained loss function, and guides the transformer architecture model training process.

[0014] Further, the training data set is generated by using the multi-level noise reduction neural network on the document corpus to be processed and the pre-configured knowledge base, and the heterogeneous graph is extracted from the training data set and the semantic entities in the document respectively, and the heterogeneous graph comprises: S31, based on the pre-configured domain knowledge base and the document corpus to be processed, the positive samples are marked, and the word level and sentence level noise reduction are processed to obtain the training data set; S32, extracting the heterogeneous graph from the training data set and the semantic entities in the document to obtain the heterogeneous graph of the training data set and the semantic entities in the document.

[0015] Further, the heterogeneous graph is fused by using the edge-oriented graph attention network model to obtain the relationship representation vector of the heterogeneous graph, and the relationship probability distribution of the relationship representation vector of the heterogeneous graph is obtained by using the multilayer perception classifier to obtain the relationship result of the heterogeneous graph entity recognition, which comprises: S41, processing the heterogeneous graph by using the edge-oriented graph attention network; S42, processing the processed heterogeneous graph by using the multi-level path attention mechanism, combining the importance of each path and assigning weights for calculation, and fusing to obtain the relationship representation of the heterogeneous graph entity; S43, inputting the relationship representation of the heterogeneous graph entity into the multilayer perception classifier to obtain the probability distribution of each type of relationship of the entity, and obtaining the relationship result of the heterogeneous graph entity recognition in the document.

[0016] Furthermore, the coreference resolution model is used to sequentially process the relationship results of the multimodal document heterogeneous graph, semantic entity, heterogeneous graph, and heterogeneous graph entity recognition, and is combined with the pre-trained link prediction model for calculation and threshold determination to obtain a global knowledge network including: S51. Build a global document graph with documents as nodes and explicit citations, topic similarities, and shared terms as edges; S52. Process the global document graph through the coreference resolution model and combine it with candidate triple verification to obtain a credible global document graph; S53. Calculate the trusted global document graph using a link prediction model to obtain a rationality score; S54. Based on a pre-set score threshold, candidate triples with scores above the pre-set threshold are deemed credible and automatically incorporated into the main knowledge graph; triples with scores below the pre-set threshold are marked as pending review and submitted for manual review or directly discarded.

[0017] Furthermore, the standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding and bounding box coordinate summation calculation formula is: ; Where, represents the final input embedding vector, Indicates the Standard word embeddings for text tokens, No. The one-dimensional position embedding of the tokens, Indicates the The 2D layout position embedding of tokens, Indicates the The logical role embedding of tokens, Indicates the The bounding box coordinates of the token on the document page, represents the position or index of a token in the input sequence, represents the text token itself, Indicates the logical role of the text block to which the i-th token belongs, Represents the i-th text token.

[0018] Furthermore, the formula for calculating the importance of each path and assigning weights to the processed heterogeneous graph using the multi-level path attention mechanism is as follows: ; ; Where, Indicates the The attention weight of each path, denotes an exponential function, N denotes the total number of paths connecting any two entity mentions in a document, n denotes an index variable in the denominator summation formula, denotes the importance of each path, denotes an independent vector representation, denotes all paths, denotes the final relationship.

[0019] The beneficial effects of the present application are: 1) The present application analyzes the document page through an advanced visual language model (LayoutLMv3), jointly processes text, image and layout information, identifies basic layout elements such as text blocks, tables, images, etc., and innovatively assigns logical role labels such as title (i.e. title), sectionHeading (i.e. chapter title), footnote (i.e. footnote), listItem (i.e. list item). For complex elements such as tables and mathematical formulas identified, special deep learning models are called for fine-grained structure recognition. For example, a semantic segmentation model based on fully convolutional network (FCN) is used to analyze the row and column structure of the table, cross-row and cross-column cells, and a parser based on context-free grammar (CFG) is used to convert formula images into MathML (i.e. Mathematical Markup Language) structured representation. All the elements and attributes (content, position, logical role, internal structure) parsed are integrated into a unified, rich multi-modal document heterogeneous graph, providing an input containing deep structure information for subsequent knowledge extraction tasks, and taking layout information such as font, font size, position as model input to improve the accuracy of entity recognition.

[0020] 2) The application adopts a pre-trained visual language model (LayoutLMv3) as a basic architecture. For each text token, the input representation is the fusion of text semantic embedding, one-dimensional sequence position embedding, two-dimensional spatial coordinate embedding, and the aforementioned logical role embedding. The model can simultaneously perceive the meaning, order, position, and function of a word. Low-rank adaptation (Low-Rank Adaptation, LoRA) technology is used for efficient parameter fine-tuning. By injecting a small number of trainable low-rank matrices into specific layers of the model (such as the query, key, and value projection matrices of the self-attention module and the newly added layout-related embedding layer), the method achieves efficient adaptation to new domain standards while freezing most pre-trained parameters. It can reduce the number of trainable parameters by more than 90%, and effectively avoid overfitting when fine-tuning on small sample data. At the same time, attention dropout (Attention Dropout) and L2 weight decay regularization techniques are introduced during the fine-tuning process to further improve the stability and generalization ability of the model. The specific model for performing deep semantic entity extraction task. The model is a LayoutLMv3-base Transformer model (i.e. a Transformer architecture model) configured for the Token Classification task. The regularization technique is applied during the process of efficient parameter fine-tuning (LoRA) of the LayoutLMv3 model (i.e. a multi-modal document understanding model), which aims to prevent the model from overfitting on a small amount of professional domain labeled data, thereby improving performance on new data. Automatically generate labeled data using domain rules and patterns to achieve relation extraction in the case of scarce labeled data.

[0021] 3) The application uses a framework to add a key multi-level noise reduction preprocessing step before model training, inspired by the idea of multi-level noise reduction neural network (MLNRNN). First, at the word level, an iterative keyword semantic aggregator (IKSA) is used to analyze each candidate relationship sentence, automatically identifying and reducing the weight of noise words unrelated to the target relationship. At the sentence level, all sentences containing the same entity pair are treated as a bag, and the multi-target multi-instance learning (MOMIL) algorithm is applied to identify the true positive sentence cluster that best represents the target relationship from the bag, and the remaining sentences are treated as false positives or noise samples. Through this word and sentence dual noise reduction mechanism, a significantly purer and more reliable training dataset can be provided for the subsequent relationship extraction model, thereby significantly improving the extraction accuracy of the model and accurately identifying and structuring complex structures such as tables, formulas, and lists.

[0022] 4) This invention constructs the input for relation extraction based on a heterogeneous graph representation of documents, comprising nodes such as entities, sentences, chapters, and table cells, along with multiple dependency edges between them. Traditional graph neural networks (GNNs) focus on updating node representations, while the core of the EoGAT model (Edge-Oriented Graph Attention Network) is learning edge representations. Through an iterative process, unique path representations are generated for the multiple paths that may exist between any two entities in the graph, such as syntactic dependency paths, paths within the same table row, and paths within the same chapter. Building on this, this invention introduces a multi-level attention mechanism: a path-level attention layer is used to dynamically evaluate the importance of different paths for determining specific relationship types and perform weighted fusion. For example, for performance metrics and test value relationships, the model may learn to assign higher weights to paths within the same table row. This enables the model to intelligently filter and integrate evidence from different levels of syntax, layout, and logical structure, effectively capturing complex relationships that are difficult to detect using traditional methods, enabling automated, end-to-end transformation from PDF documents to structured knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 This is a flowchart of a method for automatically extracting deep semantic entities and relationships from industry-standard documents according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention. They are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. By referring to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and the advantages of the present invention.

[0026] According to an embodiment of the present invention, a method for automatically extracting deep semantic entities and relationships from industry-standard documents is proposed.

[0027] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, the method for automatically extracting deep semantic entities and relationships from industry-standard documents according to an embodiment of the present invention includes: Step S1: inputting the document page image to be processed into the multimodal document understanding model for processing to obtain a multimodal document heterogeneous graph; Step S2, the multi-modal document heterogeneous graph and the preset entity type are processed through the transformer architecture model, embedding vectors of the multi-modal document heterogeneous graph and the preset entity type are extracted, and semantic entities in the document are obtained. The model of document parsing refers to a plurality of models, including LayoutLMv3 (i.e., a multi-modal document understanding model) for layout analysis, a fully convolutional network (FCN) model for table recognition, and the like.

[0028] Step S3, training data sets are generated for the document corpus to be processed and the pre-configured knowledge base by using a multi-level noise reduction neural network, and heterogeneous graphs are extracted for the training data sets and the semantic entities in the document, respectively, to obtain the heterogeneous graphs. The model of entity extraction refers to a LayoutLMv3-baseTransforme model (i.e., a transformer architecture model) for entity extraction.

[0029] Step S4, the heterogeneous graphs are fused by using an edge-oriented graph attention network model to obtain relationship representation vectors of the heterogeneous graphs, and the relationship representation vectors of the heterogeneous graphs are subjected to relationship probability distribution by a multi-layer perception classifier to obtain relationship results of the heterogeneous graph entity recognition. The model of relationship extraction refers to an edge-oriented graph attention network proposed in the present application.

[0030] Step S5, the multi-modal document heterogeneous graph, the semantic entities, the heterogeneous graphs, and the relationship results of the heterogeneous graph entity recognition are processed in sequence by using a coreference resolution model, and a pre-trained link prediction model is combined for calculation and threshold determination to obtain a global knowledge network.

[0031] The model of knowledge proofreading refers to a link prediction (LinkPrediction) model, and the model refers to a LayoutLMv3 model for entity extraction. In the embodiments, the general concept of the term "model" is consistent, but the specific objects vary with the tasks of the technical modules, and clear definitions are given in the embodiments above and below.

[0032] In this optional embodiment, the processing of the document page image to be processed into the multi-modal document understanding model to obtain the multi-modal document heterogeneous graph includes: S11, by integrating the text and image processing capabilities, the document page image to be processed is input into the multi-modal document understanding model to recognize all text blocks and visual elements on the page, and a processed document page image is obtained; S12, the processed document page image is recognized by a fully convolutional network to obtain a document page chart, and structural recognition and content association are combined to obtain structural data of the document page chart; S13, semantic parsing of the structural data is performed by using an attention mechanism encoder-decoder model and a context-free grammar parser to obtain a semantic representation of the structural data. S14, the processed document page image, the document page table, the structural data of the document page graph, and the semantic representation of the structural data are output and integrated to construct a multi-modal document heterogeneous graph.

[0033] In this optional embodiment, the multi-modal document understanding model comprises: The text and function are extracted and classified, and pre-training is performed on a large number of documents, and the multi-modal document understanding model predicts a logical role for each text block; The logical roles include title, section heading, footnote, page header, page footer, page number, and list item.

[0034] Specifically, a multi-modal document understanding model, LayoutLMv3 in specific embodiments, is used as the core engine for document layout analysis. Unlike traditional methods that rely on optical character recognition (OCR), LayoutLMv3 can uniformly process information in three modalities: text, image, and layout.

[0035] When a document page image is input, the LayoutLMv3 model performs the following operations: Content extraction: through integrated text and image processing capabilities, the model identifies all text blocks and visual elements on the page, such as pictures and dividing lines.

[0036] Geometric positioning: accurate bounding box coordinates (x1, y1, x2, y2) are generated for each identified text block, where (x1, y1) is the top-left corner coordinate and (x2, y2) is the bottom-right corner coordinate.

[0037] Logical role classification: the model extracts text and classifies functions. Using the knowledge learned from pre-training on a large number of documents, the model predicts a logical role for each text block. These roles include but are not limited to: title, section heading, footnote, page header, page footer, page number, and list item.

[0038] The step of predicting a logical role is: Obtain candidate facts: extract a new knowledge triple from the document, such as "entity1 less than entity2 greater than", and the triple is not directly stored in the final knowledge graph, but is treated as a candidate fact.

[0039] Call link prediction model: input the candidate triplets into a pre-trained TransE link (link prediction model) prediction model on the trusted knowledge graph.

[0040] Compute plausibility score: the link prediction model computes a plausibility score for the candidate triplets. The score reflects the compatibility or consistency of the triplet with the existing knowledge in the knowledge base. For example, if there are already a large number of triples in the knowledge base that state that device A is smaller than device B, and that the measurement of performance indicator X is greater than the fact, then a newly extracted triplet that states that device B is smaller than device A, and that the measurement of performance indicator X is greater than the fact will receive a higher plausibility score.

[0041] Decision and fusion: compare the computed plausibility score with a pre-set threshold.

[0042] If the score is higher than the threshold, the triplet is considered to be trusted and will be automatically incorporated into the main knowledge graph.

[0043] If the score is lower than the threshold, the triplet is labeled as pending review and can be submitted to domain experts for manual review or discarded directly, preventing the contamination of the knowledge base with false knowledge.

[0044] The specific steps are as follows: Obtain candidate facts: when the system extracts a new knowledge triplet from a document, the triplet is not directly stored in the final knowledge graph, but is considered to be a candidate fact.

[0045] Call link prediction model: input the candidate triplets into a pre-trained TransE link prediction model (i.e., link prediction model) on the trusted knowledge graph.

[0046] The above output is a series of structured page objects, each containing text content, geometric position, and logical role triple information. The introduction of logical roles provides crucial clues for understanding the macro structure and semantic hierarchy of the document.

[0047] When one of the above layout elements is identified as a table, the system passes the image region to a specialized Table Structure Recognition (TSR) submodule.

[0048] The specific implementation is as follows: Model architecture: a Fully Convolutional Network (FCN) is used, which is trained to perform pixel-level classification on table images.

[0049] Segmentation Goal: The network outputs multiple segmentation masks, each corresponding to a type of structural element, such as row separator, column separator, and header region.

[0050] Structure Reconstruction: Post-processing of the segmentation masks output by the FCN. By analyzing the connected components and relative positions of these masks, the complete grid structure of the table can be reconstructed. The algorithm can identify rows and columns and accurately handle complex rowspan and columnspan cells.

[0051] Content Association: The text content recognized by OCR is accurately filled into the reconstructed table grid according to the coordinates, and the header cells are associated with the corresponding data cells.

[0052] The table is converted into a graph-structured data containing row and column structures, cell content, and header relationships, rather than a simple text block.

[0053] For regions in the document identified as mathematical formulas, a two-stage parsing process is used to convert from image format to semantic structure representation.

[0054] Symbol Sequence Recognition: Using an encoder-decoder model based on attention mechanism (i.e., a convolutional neural network encoder plus a Transformer decoder), the formula image input is output as a corresponding symbol sequence, LaTeX string.

[0055] Syntax Structure Parsing: The LaTeX string generated in the previous step is input into a syntax analyzer based on Context-Free Grammar (CFG). The analyzer constructs a linear symbol sequence into an Abstract Syntax Tree (AST) according to predefined mathematical syntax rules. This tree clearly defines the operation structure of the formula, including operators, operands, and subscript relationships. Finally, the standardized LaTeX is output, which can be understood by machines as a mathematical meaning.

[0056] The above processing process is collectively referred to as document table structure and semantic parsing processing.

[0057] This module integrates the outputs of all the above steps to construct a global, multi-modal document heterogeneous graph = (V, E).

[0058] Node Set (V): The types of nodes are heterogeneous, including Paragraph, Heading, Table, TableCell, Formula, Figure, etc. Each node stores corresponding attributes, such as text content, bounding box, logical role, etc.

[0059] Edge Set (E): The types of edges are also diverse, representing the relationships between different elements: ReadingOrder: Connects consecutive text blocks, representing the reading order of the document.

[0060] HierarchicalContainment: Represents the logical containment relationship, for example, the TableCell node points to the Table node it belongs to, and the Paragraph node points to the Heading node it belongs to.

[0061] ExplicitReference: Represents the explicit reference relationship, for example, the Paragraph node points to the corresponding Table node.

[0062] Heterogeneous Graph is the unified input of all subsequent knowledge extraction modules of the invention, preserving the rich structure and semantic information of the document in a way far beyond pure text.

[0063] In this optional embodiment, the processing of the multi-modal document heterogeneous graph and the preset entity type through the transformer architecture model to extract the embedding vectors of the multi-modal document heterogeneous graph and the preset entity type to obtain the semantic entities in the document includes: S21, taking the multi-modal document heterogeneous graph and the preset entity type as core input data, extracting embedding vectors of the multi-modal document heterogeneous graph and the preset entity type through standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding, and boundary box coordinate addition calculation; S22, processing the extracted embedding vectors through low-rank adapter and regularization technology in turn to obtain the semantic entities in the document.

[0064] In this optional embodiment, the regularization technology includes: attention dropout and weight decay; Among them, the attention dropout inputs the extracted embedding vectors into the attention calculation to obtain the attention scores, selects part of the attention scores to be zero by random selection, and inputs the attention scores into the normalization function for calculation; Weight decay, determining the loss function in the training of the transformer architecture model and the weight parameters of the low-rank adaptive matrix, adding a penalty term proportional to the norm square of the weight of the low-rank adaptive matrix to the initial loss function to generate the trained loss function, and guiding the transformer architecture model training process.

[0065] Specifically, to solve the problem of fine-tuning on a small amount of labeled data, low-rank adaptive (LoRA) fine-tuning: through LoRA technology, during training, the vast majority of the original weights of the pre-trained model are frozen unchanged, and only small, trainable "adapter" modules are injected into specific layers of the Transformer architecture. Specifically, for a weight matrix , the update amount is decomposed into the product of two low-rank matrices , where and , and the rank is much smaller than the original dimension and During fine-tuning, only the parameters of matrices A and B are trained. Decomposed into the product of two smaller (low-rank) matrices.

[0066] and is a standard mathematical symbol representing the set of real numbers. Each element in matrix A is a real number. In the field of machine learning and deep learning, it is a very common representation method because the weights of the model (such as matrix A) are usually represented by real numbers.

[0067] is the original weight matrix in the pre-trained model that is frozen during the fine-tuning process.

[0068] is the weight update amount that needs to be learned during the fine-tuning process. The final model weight is .

[0069] B and A are two low-rank matrices that make up the update amount , that is, During the fine-tuning process, only the parameters of these two small matrices A and B are trained, rather than the entire huge matrix.

[0070] is the rank of low-rank decomposition. A hyperparameter much smaller than the original matrix dimension (in the embodiment of the present invention, ). The value of determines the number of trainable parameters and is key to efficient parameter fine-tuning. In this embodiment, MLNRNN stands for Multi-level Noise Reduction Neural Network. This illustrates how multi-level noise reduction preprocessing in the weakly supervised relation extraction framework identifies and suppresses noise at different granularities, such as the word level and sentence level.

[0071] The selection of LoRA target modules depends on the application location of LoRA adapters (i.e. low-rank adapters). To maximize the synergy between text and layout information, LoRA is strategically applied to the following modules: The query, key, and value projection layers in the self-attention mechanism. A new two-dimensional position embedding layer ( ) and the logical role embedding layer ( This design forces the model’s adaptation to focus on learning how to adjust attention patterns based on text and layout / character cues, achieving a deep and efficient fusion of information from both modalities.

[0072] Regularization technology: In the LoRA fine-tuning process, to prevent overfitting and improve model robustness, this paper integrates two regularization methods: Attention Dropout: After calculating the attention weights and before applying the softmax function, a portion of the attention scores are randomly set to zero. This prevents the model from over-relying on a few attention heads and promotes more diverse attention patterns.

[0073] L2WeightDecay: Adds a penalty term to the loss function that is proportional to the square of the L2 norm of the weights of the LoRA matrices (A and B). This effectively prevents the weights from becoming too large, thereby improving the generalization ability of the model. As shown in Table 1: Table 1 Implementation parameters

[0074] In this optional embodiment, the multi-stage noise reduction neural network is used to generate a training data set for the document corpus to be processed and the pre-configured knowledge base, and a heterogeneous graph is extracted from the semantic entities in the training data set and the document respectively, and the obtained heterogeneous graph includes: S31. Based on the pre-configured domain knowledge base and the document corpus to be processed, mark them as positive samples and process them through word-level and sentence-level noise reduction to obtain a training dataset; S32, extract the heterogeneous graph of semantic entities in the training data set and the document, to obtain the heterogeneous graph of semantic entities in the training data set and the document.

[0075] Specifically, two core problems of relation extraction are solved, how to reduce the dependence on manual annotation, and how to accurately extract complex relationships across long text distances. The multi-level noise reduction training data generation of remote supervision: In order to automatically generate training data for relation extraction, a remote supervision method is adopted, but a key noise suppression mechanism is introduced to solve the inherent defects of the hypothesis.

[0076] Initial data annotation, i.e. remote supervision: use an existing domain knowledge base or ontology, for example, a database containing standard part parameters and relationships. For each relationship triple in the knowledge base, entity1, relationship, entity2, search all sentences containing entity1 and entity2 in the document corpus to be processed. The sentence is preliminarily and automatically labeled as a positive sample expressing the relationship.

[0077] Multi-level noise reduction: before using automatically annotated data for model training, a multi-level noise reduction neural network is used to identify and suppress noise at different granularities, such as word level and sentence level, in the multi-level noise reduction preprocessing step of the weakly supervised relation extraction framework.

[0078] Word-level noise reduction: for each preliminarily labeled sentence, an iterative keyword semantic aggregator (IKSA) is used for processing. IKSA calculates an importance weight for each word by analyzing the semantic relevance of the word to the target entity pair and the relationship. Context words unrelated to relationship expression, such as in addition, however, etc. will be given a lower weight. This is used to filter out irrelevant information within the sentence and highlight the key words that actually carry relationship information.

[0079] Sentence-level noise reduction: all sentences containing the same entity pair are considered as a bag. The bag is filtered by applying a multi-target multi-instance learning (MOMIL) algorithm. The goal of MOMIL is to identify one or more true positive sentences within the bag that can clearly and unambiguously express the target relationship. This is achieved by clustering or similarity calculation in the feature space, identifying sentences containing the entity pair but not actually expressing the relationship (i.e. noise) and removing or reducing the weight of the sentences from the training set.

[0080] Through the two-step noise reduction process of pollution first and then governance, a much higher quality training data set is generated than the traditional remote supervision method, laying a solid foundation for training high-precision relation extraction models.

[0081] Heterogeneous graph construction for relation extraction, for each document, a heterogeneous graph for relation extraction is constructed = (V, E), the structure of the heterogeneous graph is more detailed than the document graph in module one, focusing on the connections between entities.

[0082] Node (V): Mention: represents a specific mention of an entity in the document.

[0083] Entity: represents an abstract entity, and all Mention nodes of the same entity point to E.

[0084] Sentence: represents a sentence, and Mention nodes are linked to the corresponding Sentence nodes.

[0085] Section: represents a chapter or section, and Sentence nodes are linked to the corresponding Section nodes.

[0086] TableCell: if the entity mention appears in a table, the Mention node is linked to the corresponding TableCell node.

[0087] Edge (E): Containment: represents various containment relationships, such as mention sentence, sentence chapter, etc.

[0088] Co-occurrence: connects Mention nodes that appear in the same sentence / section / table row.

[0089] SyntacticDependency: based on the syntactic dependency analysis tree, connects Mention nodes corresponding to word pairs with direct syntactic relationships within a sentence.

[0090] Coreference: connects different Mention nodes that refer to the same entity 0, such as device and spectrum analyzer.

[0091] Edge-oriented graph attention network and multi-level attention, to realize cross-sentence and even cross-paragraph relationship extraction, edge-oriented graph attention network.

[0092] Edge-oriented paradigm: existing graph attention networks (GAT) update the representation of the center node by aggregating the information of neighboring nodes. The relationship between entities is more reflected in the connection path. Adopting an edge-oriented (edge-oriented) paradigm, the core computing unit of the model is to update the representation of the edge, rather than the node. The representation of an edge is iteratively updated based on the information of the two connected nodes and the adjacent edges.

[0093] Multi-path representation learning: in the heterogeneous graph , any two entity mentions and There can be multiple paths of different types between Syntactic paths: greater than subject-predicate relationship greater than v greater than verb-object relationship greater than .

[0094] Table paths: greater than greater than same row greater than greater than .

[0095] Chapter co-occurrence paths: greater than greater than greater than greater than The EoGAT model learns an independent vector representation for each such path , which encodes the structural and semantic information of the path.

[0096] Multi-level path attention mechanism: This is the core mechanism designed by the present application to solve complex relation extraction. After obtaining all the path representations between the entity pair , ). .

[0097] Iterative Keyword Semantic Aggregator (IKSA) is an algorithmic module used for word-level noise reduction before weakly supervised relation extraction.

[0098] Meaning: For a sentence that may contain the target relation, the algorithm automatically evaluates the relevance of each word in the sentence to the target relation and assigns different importance weights. The purpose is to filter out irrelevant information within the sentence and highlight the key words that truly carry relationship information.

[0099] and : These two symbols are used in this embodiment to illustrate the sentence nodes (SentenceNode) when constructing paths. For example, in the example of chapter co-occurrence paths, a path may contain the nodes where the entities are located, forming intermediate nodes connecting the two entity paths.

[0100] represent the table cell nodes (TableCellNode) when constructing paths. For example, in a table path connected through table structure, represent the entities​ the table cell node where the entity is located.

[0101] The model can adaptively learn the weights of different types of evidence paths according to the type of relationship to be predicted. For example, when predicting dependency, the model learns to give higher attention weights to syntactic paths; while when predicting belonging to the same group, it pays more attention to table paths.

[0102] Relation classification: final relation representation is fed into a multi-layer perceptron (MLP) classifier, which outputs a probability distribution over possible relations between the entity pair through a Softmax layer.

[0103] Specific analysis steps: For a candidate relation sentence, for example, a sentence containing both the entity insulation resistance and the entity test voltage, to determine whether it expresses the test condition, the analysis steps of IKSA are as follows: Input: a candidate sentence, the identified entity pair in the sentence, for example, insulation resistance, test voltage, and a target relation, for example, test condition.

[0104] Semantic relevance analysis: IKSA analyzes the relevance by calculating the semantic similarity between the vector representation of each word in the sentence and the vector representation of the target entity pair and the target relation.

[0105] Weight calculation: according to the semantic relevance score, calculate an importance weight for each word in the sentence. Words closely related to the target relation expression, for example, should be tested under, will get high weight, and irrelevant context words, for example, in addition, however, as shown in the figure, will be given a lower weight.

[0106] Iterative optimization, that is, iterative embodiment: the above process can be repeated, using the high-weight words calculated in the last round to further focus and optimize the weights of other words, until the weight distribution converges stably.

[0107] Output: a sentence representation with word weight information. The weighted sentence representation will be used for subsequent sentence-level noise reduction or relation extraction model training. Since the weights of noisy words have been reduced, the model training will be more robust.

[0108] In this optional embodiment, the heterogeneous graph is fused by using the edge-oriented graph attention network model to obtain a relation representation vector of the heterogeneous graph, and a multi-layer perceptron classifier is used to obtain a relation probability distribution of the relation representation vector of the heterogeneous graph to obtain a relation result of entity recognition of the heterogeneous graph. S41, processing the heterogeneous graph by using the edge-oriented graph attention network; S42, the processed heterogeneous graph is processed by using a multi-level path attention mechanism, and weights are assigned and calculated according to the importance of each path to obtain a relationship representation of the heterogeneous graph entity; S43, inputting the relationship representation of the heterogeneous graph entity into a multi-layer perception classifier to obtain a probability distribution of each relationship of the entity, and obtaining a relationship result of the identified heterogeneous graph entity in the document.

[0109] Specifically, the knowledge extracted from a single document is integrated to construct a consistent and accurate knowledge network covering the entire standard system.

[0110] Knowledge in the industry standard system is often scattered in multiple mutually referenced documents. In order to link the above isolated knowledge, an advanced cross-document coreference resolution (CDCR) technology is used.

[0111] Global document graph construction: a global document graph is constructed with documents as nodes and explicit references, theme similarity or shared terms as edges.

[0112] In this optional embodiment, the coreference resolution model is used to sequentially process the multi-modal document heterogeneous graph, semantic entity, heterogeneous graph and relationship result of the identified heterogeneous graph entity, and the pre-trained link prediction model is used for calculation and threshold determination to obtain a global knowledge network, including: S51, a global document graph is constructed with documents as nodes and explicit references, theme similarity and shared terms as edges; S52, the global document graph is processed by the coreference resolution model, and the candidate triple verification is combined to obtain a reliable global document graph; S53, the link prediction model is used to calculate the reliable global document graph to obtain a rationality score; S54, based on the preset score threshold, the candidate triple higher than the preset threshold is considered as reliable and automatically incorporated into the main knowledge graph; the triple lower than the preset threshold is marked as to be reviewed and submitted for manual review or directly discarded.

[0113] Specifically, based on the probabilistic knowledge correction of link prediction, in order to ensure the quality of the final knowledge in the database and prevent extraction errors from polluting the knowledge graph.

[0114] Candidate triple verification: when the system extracts a new knowledge triple from the document, less than entity 1, relationship, entity 2 greater than, the triple is not directly stored in the database and is regarded as a candidate fact.

[0115] Link prediction model: The model can be based on classic models of graph embedding or more powerful graph neural network-based models, such as TransE, ComplEx (i.e., knowledge graph embedding).

[0116] Plausibility score: The link prediction model calculates a plausibility score for the candidate triplets, which reflects the compatibility of the triplet with the existing knowledge in the knowledge base. For example, if there are a large number of facts in the knowledge base that "Device A, Measurement, Performance Indicator X is less than", then the newly extracted "Device B, Measurement, Performance Indicator X is greater than" will obtain a higher score, where Device B is similar in type to Device A.

[0117] Decision and fusion: Set a plausibility score threshold. Candidate triplets above the threshold are considered reliable and automatically incorporated into the main knowledge graph. Triplets below the threshold are marked as pending review and submitted to domain experts for manual review or discarded directly.

[0118] EoGAT model is used to solve the problem of complex relationship extraction across sentences and paragraphs.

[0119] Specific reference and inclusion content: EoGAT model is an edge-oriented graph attention network (Edge-Oriented Graph Attention Network). EoGAT model is a novel graph neural network (GNN) architecture. Unlike traditional GNNs, which focus on updating the representation of graph nodes such as entities, the core paradigm of EoGAT is edge-oriented, i.e., the core computing unit is to learn and update the representation of edges in the graph, which represents the possible relationship paths between entities.

[0120] Mainly contains three core mechanisms: Edge-oriented update paradigm: The focus of the model's calculation is to update the representation of the edge, rather than the representation of the node.

[0121] Multi-path representation learning: It can learn independent vector representations for multiple different types of paths connecting two entities (such as syntactic paths, table paths, etc.).

[0122] Multi-level path attention mechanism: Through an attention layer, dynamically evaluate and fuse the importance of different paths for judging a specific relationship type.

[0123] Specific steps based on EoGAT model: Input construction: Based on the document, build a heterogeneous graph dedicated to relationship extraction The heterogeneous graph contains multiple types of nodes such as entity mentions, sentences, chapters, table cells, and multiple dependency relationship edges between them.

[0124] Multi-path recognition and representation learning: For any two entity mentions and , the EoGAT model will identify K different paths connecting, for example, syntactic dependency paths, same-table-row paths, same-section co-occurrence paths, etc. The model will learn an independent vector representation hk for each path k, which encodes the structure and semantic information of the path.

[0125] Multi-level attention for path importance evaluation: A path-level attention module calculates each path representation The importance of the final relationship prediction. This is calculated by a learnable function to calculate the attention weight of each path The weights are dynamic; for example, when predicting the relationship between performance metrics and test values, the model may learn to give higher weights to paths with the same table row.

[0126] Path information fusion: vector representation of all paths According to the corresponding attention weight Perform weighted summation to obtain a comprehensive, representative entity The final representation vector of the relationship between the pair .

[0127] Relationship classification: The final relationship representation vector It is fed into a standard multi-layer perceptron (MLP) classifier and outputs the probability distribution of all predefined relationships between entity pairs through a Softmax layer to complete the relationship prediction.

[0128] Prediction refers to the step of using the link prediction model to evaluate and proofread the newly extracted knowledge triples in cross-document knowledge fusion and proofreading.

[0129] The specific steps are as follows: Obtaining candidate facts: After extracting a new knowledge triple from the document, for example, entity 1 is less than, relation, entity 2 is greater than, the triple is not directly stored in the final knowledge graph, but is regarded as a candidate fact.

[0130] Calling the link prediction model: Input the candidate triples into a TransE link (link prediction model) prediction model that has been pre-trained on the trusted knowledge graph.

[0131] Computing plausibility score: The link prediction model computes a plausibility score for the candidate triple. The score reflects the compatibility or consistency of the triple with the existing knowledge in the knowledge base. For example, if there are already a large number of triples in the knowledge base that state that a device A, measurement, performance metric X is less than a fact, then a newly extracted triple that states that a device B, measurement, performance metric X is less than, where device B is similar in type to device A, will get a higher plausibility score.

[0132] Decision and fusion: The computed plausibility score is compared with a pre-set threshold.

[0133] If the score is higher than the threshold, the triple is considered to be trustworthy and will be automatically incorporated into the main knowledge graph.

[0134] If the score is lower than the threshold, the triple is labeled as pending review and can be submitted to scholars for manual review or discarded directly, thus preventing false knowledge from polluting the knowledge base.

[0135] In this optional embodiment, the model is a LayoutLMv3-base Transformer model configured for a Token Classification task (i.e., a Transformer architecture model). The innovation lies in the unique input embedding layer design that deeply fuses text and layout information. For the i-th token in the input sequence , the final input embedding vector is the sum of the following four parts. The sum calculation formula of the standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding, and bounding box coordinates is: ; In the formula, represents the final input embedding vector, represents the standard word embedding of the i-th text token, represents the one-dimensional position embedding of the i-th token, represents the two-dimensional layout position embedding of the i-th token, represents the logical role embedding of the i-th token, represents the bounding box coordinates of the i-th token on the document page, represents the position or index of the token in the input sequence, represents the text token itself, represents the logical role of the i-th text token, represents the i-th text token.

[0136] ​​​​​ represents the final input embedding vector, representing the semantic information of the token itself, obtained by querying the vocabulary of the model.

[0137] represents the standard word embedding of the i-th text token, the standard word embedding obtained by querying the vocabulary of the model, representing the semantic information of the token.

[0138] represents the one-dimensional position embedding of the i-th token, representing the order information of the token in the entire text sequence.

[0139] represents the two-dimensional layout position embedding of the i-th token, representing the physical space position of the token on the document page, which is calculated according to the bounding box coordinates (bbox i ) of the token.

[0140] represents the position or index of the token in the input sequence, described as the i-th token in the document.

[0141] represents the text token (token) itself. In the formula, it refers to the standard word embedding of the i-th text token. represents the i-th text token.

[0142] represents the logical role of the text block to which the i-th token belongs. For example, this role can be a title, a list item, or a footnote, etc.

[0143] The bounding box coordinates of the token are normalized to the integer interval. Each coordinate component (left, top, width, height) is mapped to a vector by an independent trainable embedding layer, and the vectors are concatenated or added to form an embedding vector representing the absolute position of the token on the page.

[0144] represents the logical role embedding of the i-th token, representing the functional role of the text block to which the token belongs in the document structure, such as title, list item, footnote, etc., and the role information is obtained by module one.

[0145] ​​​​​Representing logical role embedding, the logical role of the text block in which the token identified by module 1 is located, such as sectionHeading (i.e., chapter title), is mapped to a trainable embedding vector. The embedding provides powerful contextual clues for the model. For example, the model can learn that text appearing in the sectionHeading role is more likely to be an important technical requirement entity.

[0146] In this way, when processing each token, the model can simultaneously obtain information about what it is (semantics), where it is (sequence), where it is on the page (space), and what type of content block it is in (function), thereby greatly enhancing the accuracy of entity recognition.

[0147] Indicates the The bounding box coordinates of a token on the document page, usually expressed as a set of coordinate values, such as the upper left corner x, y and the lower right corner x, y, precisely define the position and range of the token on the two-dimensional page.

[0148] In this optional embodiment, a path-level attention module is used to compute the final relation representation The formula for calculating the weights of the processed heterogeneous graphs by using the multi-level path attention mechanism and combining the importance of each path is as follows: ; ; Where, Indicates the The attention weight of each path, represents an exponential function, N represents the total number of paths between any two entity mentions in two entity mention documents, and n represents the index variable in the denominator summation formula. Indicates the importance of calculating each path, represents an independent vector representation, Indicates all paths, Indicates the final relationship.

[0149] Specifically, Represents an exponential function. In this formula, it is the core component of the Softmax function, which is used to map the score of each path to a positive number and widen the gap between different scores, allowing the model to more clearly focus on the most important paths.

[0150] N represents the total number of paths between any two entity mentions in the document connecting the two entity mentions. and The total number of paths between them. All possible paths connecting two entities are identified in the document's heterogeneous graph, such as syntactic paths, table paths, chapter co-occurrence paths, etc. This is the sum of these paths.

[0151] n represents the index variable in the denominator summation formula, which traverses from 1 to N, representing each specific path.

[0152] represents the importance of each path, which is a small feedforward network for calculating the importance of each path.

[0153] The above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for automatic extraction of deep semantic entities and relations for industry standard documents, characterized in that, The method comprises the following steps: S1, input the document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph; S2, process the multi-modal document heterogeneous graph and the preset entity type through a transformer architecture model to extract the embedding vectors of the multi-modal document heterogeneous graph and the preset entity type, and obtain the semantic entity in the document; S3, use a multi-level noise reduction neural network to generate a training data set from the document corpus to be processed and a pre-configured knowledge base, and extract a heterogeneous graph from the training data set and the semantic entity in the document, respectively, to obtain a heterogeneous graph; S4, use an edge-oriented graph attention network model to fuse the heterogeneous graph to obtain a relationship representation vector of the heterogeneous graph, and use a multi-layer perception classifier to perform relationship probability distribution on the relationship representation vector of the heterogeneous graph to obtain a relationship result of entity recognition of the heterogeneous graph; S5, use a co-reference resolution model to process the multi-modal document heterogeneous graph, the semantic entity, the heterogeneous graph, and the relationship result of entity recognition of the heterogeneous graph in sequence, and combine a pre-trained link prediction model to perform calculation and threshold judgment to obtain a global knowledge network.

2. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The method comprises the following steps: S11, input the document page image to be processed into a multi-modal document understanding model for processing to obtain a multi-modal document heterogeneous graph; S12, identify the document page image after processing through a full convolutional network to obtain a document page chart, and combine structure recognition and content association to obtain structure data of the document page graph; S13, use an attention mechanism encoder-decoder model and a context-free grammar parser to perform semantic parsing on the structure data to obtain a semantic representation of the structure data; S14, output and integrate the document page image after processing, the document page chart, the structure data of the document page graph, and the semantic representation of the structure data to construct a multi-modal document heterogeneous graph.

3. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 2, characterized in that, The multi-modal document understanding model comprises: extract text and function for classification, and pre-train using knowledge on a large number of documents, the multi-modal document understanding model predicts a logical role for each text block; the logical roles include title, chapter title, footnote, header, footer, page number, and list item.

4. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The method comprises the following steps: S21, input the multi-modal document heterogeneous graph and the preset entity type as core input data, and calculate the embedding vectors of the multi-modal document heterogeneous graph and the preset entity type through standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding, and boundary box coordinate addition; S22, process the extracted embedding vectors through a low-rank adapter and a regularization technique in sequence to obtain the semantic entity in the document.

5. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 4, characterized in that, The regularization technique comprises: attention dropout and weight decay; Among them, attention dropout, input the extracted embedding vector into the attention score calculated by attention calculation, part of the attention score is zero by random selection, and the attention score is input into the normalization function for calculation; Weight decay, determine the loss function and the weight parameter of the low rank adaptive matrix in the transformer architecture model training, add a penalty term proportional to the square norm of the weight of the low rank adaptive matrix to the initial loss function, generate the trained loss function, and guide the transformer architecture model training process.

6. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The training data set is generated by using the multi-level noise reduction neural network on the document corpus to be processed and the pre-configured knowledge base, and the semantic entities in the training data set and the document are extracted respectively The heterogeneous graph is obtained, including: S31, based on the pre-configured domain knowledge base and the document corpus to be processed, mark as positive samples, process by word level and sentence level noise reduction, obtain training data set; S32, extract the heterogeneous graph of the semantic entity in the training data set and the document, and obtain the heterogeneous graph of the semantic entity in the training data set and the document.

7. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 5, characterized in that, The relationship representation vector of the heterogeneous graph is obtained by using the edge directed graph attention network model to fuse the heterogeneous graph, and the relationship probability distribution of the relationship representation vector of the heterogeneous graph is obtained by using the multilayer perception classifier, and the relationship result of the entity recognition of the heterogeneous graph is obtained, including: S41, using edge directed graph attention network to process the heterogeneous graph; S42, using multi-level path attention mechanism and processing the processed heterogeneous graph, combining the importance of each path and assigning weights for calculation, and fusing to obtain the relationship representation of the entity of the heterogeneous graph; S43, input the relationship representation of the entity of the heterogeneous graph into the multilayer perception classifier to obtain the probability distribution of each type of relationship of the entity, and obtain the relationship result of the entity recognition of the heterogeneous graph in the document.

8. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 1, characterized in that, The relationship result of the entity recognition of the heterogeneous graph is obtained by using the co-reference resolution model to process the multi-modal document heterogeneous graph, semantic entity, heterogeneous graph and heterogeneous graph in turn, and combining the pre-trained link prediction model for calculation and threshold determination, and obtaining the global knowledge network, including: S51, construct a global document graph taking document as node and explicit reference, theme similarity and shared term as edge; S52, process the global document graph by the co-reference resolution model, and combine the candidate triple verification to obtain the reliable global document graph; S53, use the link prediction model to calculate the reliability score of the reliable global document graph; S54, based on the preset score threshold, the candidate triple higher than the preset threshold is considered as reliable and automatically incorporated into the main knowledge graph; The triple lower than the preset threshold is marked as to be reviewed and submitted for manual review or directly discarded. 9.The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 4, characterized in that, The standard word embedding, one-dimensional position embedding, two-dimensional layout position embedding, logical role embedding and boundary box coordinate addition calculation formula are: ； wherein, denotes the final input embedding vector, denotes the standard word embedding of the th text token, denotes the one-dimensional position embedding of the th token, denotes the two-dimensional layout position embedding of the th token, denotes the logical role embedding of the th token, denotes the bounding box coordinates of the th token on the document page, denotes the position or index of the token in the input sequence, denotes the text token itself, denotes the logical role of the text block to which the i-th token belongs, denotes the i-th text token.

10. The industry standard document oriented deep semantic entity and relation automatic extraction method according to claim 7, characterized in that, The formula for processing the processed heterogeneous graph by using the multi-level path attention mechanism, combining the importance of each path and assigning weights for calculation is: The formula for processing the processed heterogeneous graph by using the multi-level path attention mechanism, combining the importance of each path and assigning weights for calculation is: ； ； wherein, denotes the attention weight of the path, denotes the exponential function, N denotes the total number of paths connecting any two entity mentions in a document, n denotes the index variable in the denominator summation formula, denotes the importance of each path, denotes the independent vector representation, denotes all paths, denotes the final resulting relation.

Images