Intelligent checking method for work ticket based on multi-modal large model knowledge graph and large model reasoning

Abstract

The work ticket intelligent checking method based on multi-modal large model knowledge graph and large model reasoning belongs to the field of power system application technology, first extracts text feature vectors and image feature vectors through an OCR module and an image encoder, forms a comprehensive semantic representation after multi-modal fusion. Extract entities and relationships from power safety regulations and historical operation data to generate a power safety knowledge graph. Check the format and integrity of the basic fields on the ticket, use LLM to combine context and graph nodes for semantic reasoning to generate semantic judgment results. Fuse three layers of results to calculate a comprehensive checking score and output an intelligent determination result. The present application introduces multi-modal input analysis, knowledge graph construction and large model semantic reasoning technology to construct a structured, interpretable and iteratively optimized intelligent work ticket checking system that realizes the full-process intelligentization from ticket image recognition to semantic understanding and from explicit rule checking to implicit logical reasoning.

Description

Technical Field

[0001] This invention belongs to the field of power system application technology, specifically relating to an intelligent verification method for work tickets based on multimodal large model knowledge graph and large model reasoning. Background Technology

[0002] With the digital transformation of power systems, the management and safety auditing of on-site operations have placed higher demands on automation and intelligence. Work orders, as crucial documents in power maintenance and operation processes, include work tasks, isolation measures, safety measures, equipment information, and operating procedures, serving as the core basis for ensuring safe execution on-site.

[0003] Current work order verification mainly suffers from the following problems: Work order issuers need to manually fill in the work content, isolation measures, and safety measures, which can easily lead to errors such as incorrect equipment names, non-standard terminology, and missing fields; work permit issuers rely on personal experience for verification, and in complex work scenarios (especially secondary equipment maintenance involving multi-disciplinary collaboration), they are prone to overlooking key elements or making logical inconsistencies; existing tools mostly remain at the rule matching level, lacking the ability to understand semantics incorporating power knowledge, making it difficult to achieve comprehensive analysis of text content, topology maps, and real-time operating conditions; current verification systems are mostly based on plain text input, failing to process unstructured data such as PDF scans and images, and cannot support automatic parsing and semantic understanding of document images.

[0004] Therefore, an intelligent verification method based on multimodal understanding and knowledge enhancement is needed to intelligently parse scanned PDFs and image-based work orders and verify their content compliance, achieving end-to-end automated processing from text recognition and semantic understanding to security measure verification. Summary of the Invention

[0005] The purpose of this invention is to propose an intelligent verification method for work tickets based on multimodal large-scale model knowledge graphs and large-scale model reasoning. By introducing multimodal input parsing, knowledge graph construction and large-scale model semantic reasoning technologies, a structured, interpretable and iteratively optimizable intelligent work ticket verification system is constructed. This system realizes intelligent processing from ticket image recognition to semantic understanding, and from explicit rule verification to implicit logic reasoning.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for intelligent verification of work tickets based on multimodal large-scale model knowledge graphs and large-scale model reasoning, comprising the following steps:

[0008] Step 1: Multimodal Input Parsing: The system receives scanned PDF files and on-site images, extracts text feature vectors and image feature vectors through the OCR module and image encoder respectively, and forms a comprehensive semantic representation after multimodal fusion.

[0009] Step 2: Knowledge Graph Construction: The system extracts entities and relationships from power safety regulations and historical operation data to generate a power safety knowledge graph, which is then stored and indexed in a graph database to provide structured knowledge support for subsequent reasoning.

[0010] Step 3, Layered Verification: The basic fields of the ticket are validated for format and completeness; logical matching scores are calculated based on the graph relationship chain to identify potential security measure conflicts or omissions; semantic reasoning is performed using LLM combined with context and graph nodes to generate semantic judgment results. The system ultimately integrates the three layers of results, calculates a comprehensive verification score, and outputs an intelligent judgment result.

[0011] Step 4: Output Generation and Feedback: The verification conclusions are returned in the form of a structured report, PDF export, or system interface, enabling real-time feedback and dynamic updates of the knowledge graph.

[0012] The core technical key points of this invention are mainly reflected in:

[0013] Key Technical Point 1: Integration of Multimodal Semantic Understanding and Entity Extraction

[0014] By fusing OCR, visual models, and semantic encoders, structured entities and semantic features can be automatically extracted from scanned tickets, laying the foundation for subsequent map construction.

[0015] Key Technical Point Two: Structured Retrieval and Reasoning Based on Knowledge Graphs

[0016] Knowledge graphs explicitly express power regulations, equipment relationships, and safety measures, supporting logical verification and semantic consistency validation based on relationship chains, thus upgrading from "semantic matching" to "knowledge reasoning".

[0017] Key Technical Point 3: Multi-layer Semantic Fusion Determination Mechanism

[0018] By adopting a hierarchical verification architecture of "rule layer + graph layer + large model layer", the entire process of verification, from explicit error detection to deep logical consistency judgment, is realized, thereby improving accuracy and interpretability.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] ① This invention can automatically parse work orders such as scanned PDFs, input images, and on-site scene images, integrate multimodal information such as text and images, automatically identify key fields such as work tasks, safety measures, and operation steps, realize the structural and semantic conversion of documents, and reduce manual input steps.

[0021] ② This invention, through entity relationship modeling and graph reasoning mechanism based on knowledge graph, can automatically determine whether safety measures, equipment isolation, grounding operations, etc., comply with the State Grid safety regulations, realize the consistency verification of operation logic, and significantly reduce the risks and hidden dangers caused by non-standard filling.

[0022] ③ The three-layer structure of "rule matching layer - graph retrieval layer - large model judgment layer" proposed in this invention effectively combines explicit rule detection and deep semantic understanding, which not only ensures the comprehensiveness of the verification, but also improves the intelligence level of semantic judgment. The overall judgment accuracy is improved by more than 20% compared with traditional rule-based or text vector-based methods.

[0023] ④ This invention can automatically expand the knowledge graph based on new regulations and the latest work order data, realize incremental learning and adaptive updates, and enable the verification model to be continuously optimized in long-term operation and keep in sync with actual safety management standards. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of the intelligent verification method for work tickets based on multimodal large model knowledge graph and large model reasoning proposed in this invention.

[0025] Figure 2 This is a detailed explanation diagram of each step of the work ticket intelligent verification method based on multimodal large model knowledge graph and large model reasoning proposed in this invention. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0027] like Figure 1 and 2 As shown, this embodiment proposes an intelligent verification method for work tickets based on a multimodal large-scale model knowledge graph and large-scale model reasoning. The input includes scanned PDF and image-format work tickets, and the output is a structured verification report and modification suggestions. The entire process is divided into four stages, realizing an automated closed-loop verification system from document image parsing to semantic reasoning, and from rule matching to logical inference, significantly improving the intelligence, accuracy, and interpretability of power work ticket verification. Detailed explanations of each step are as follows:

[0028] Step 1: In the multimodal input parsing stage, the system receives scanned PDF files and on-site images, and extracts text feature vectors through the OCR module and image encoder, respectively. With image feature vectors After multimodal fusion, a comprehensive semantic representation is formed. The detailed steps are as follows:

[0029] ① Perform OCR recognition on the input scanned PDFs and images to extract text content and table structure, resulting in a set of text paragraphs:

[0030]

[0031] in, It is the collection of text paragraphs or fields obtained after OCR parses the entire document. It is a set The first in A text paragraph or field such as "Work Tasks" or "Safety Measures", It is the total number of text paragraphs or fields, i.e., the set. The number of elements in the text.

[0032] ② Embed the image region into an image encoder (such as CLIP, VIT, or NFNet) to generate visual feature vectors:

[0033]

[0034] in, It is the first Image feature vectors of a single image or image region These are image coding models (such as CLIP, ViT, etc.) used to map images to vector space. Indicates the first Input an image or image region, such as a photo of a work order or a tile on a ticket page.

[0035] ③ Encode text paragraphs into semantic vectors:

[0036]

[0037] in, It is related to the text segment The corresponding text semantic vector is the text encoding result. This refers to text encoding models (such as BERT, Sentence-BERT, etc.) used to map text to a vector space.

[0038] ④ Use a multimodal fusion model to obtain a comprehensive feature representation:

[0039]

[0040] in, Indicates the first Each entry is a multimodal fusion semantic representation vector that integrates text and image information. The fusion weights are used to balance the contributions of text and image features.

[0041] Step 2: In the knowledge graph construction phase, the system extracts entities and relationships from power safety regulations and historical operation data to generate a power safety knowledge graph. The data is then stored and indexed in a graph database to provide structured knowledge support for subsequent reasoning. The detailed steps are as follows:

[0042] This invention no longer uses traditional vector libraries, but instead constructs a knowledge graph that covers multiple sources of information, including power safety regulations, equipment parameters, historical invoices, and risk measures.

[0043] First, we define a knowledge graph as a set of triples:

[0044]

[0045] in:

[0046] : A collection of entities (including equipment, personnel, measures, procedures and clauses, operation tickets, etc.);

[0047] : Sets of relations (such as "contains in", "applies to", "causes", "constrains")

[0048] : Represents a fact triple that states "there is a relation r between the head entity h and the tail entity t".

[0049] The construction process includes:

[0050] Entity extraction: Key objects in the invoice text, such as "transformer", "grounding switch", and "work supervisor", are identified using the Named Entity Recognition (NER) algorithm.

[0051] Relation extraction: Extracting logical relationships between entities based on dependency parsing and template rules;

[0052] Attribute fusion and deduplication: Align entities with the same name or synonyms to build a unified entity library;

[0053] Graph structure storage: Utilize graph databases (such as Neo4j) to store graph nodes and edges, and support query and reasoning operations.

[0054] Knowledge graphs support calculating semantic similarity through relational chains:

[0055]

[0056] in This indicates the path length considered during matching. Indicates the index of the relationship chain. For relation matching functions, targeting the first... Relationships contained in the path If the relationship matches the expectation, a higher value will be obtained. This is the path weight, used to measure the importance of the path.

[0057] Step 3, in the layered verification mechanism stage: Perform format and completeness checks on the basic fields of the ticket; calculate the logical matching score based on the graph relationship chain. Identify potential security measure conflicts or omissions; utilize LLM to perform semantic reasoning by combining context and graph nodes to generate semantic judgment results. The system ultimately integrates the results from the three layers to calculate a comprehensive verification score. It outputs intelligent judgment results.

[0058] To achieve high accuracy and interpretability, this invention employs a three-layer verification mechanism, analyzing from shallow to deep level, as detailed below:

[0059] First layer: Rule matching layer

[0060] Based on regular expressions and predefined rule sets Basic validation of key fields on the ticket, including:

[0061] ① Check if any fields are missing (such as "Work Supervisor" or "Safety Measure Number");

[0062] ② Verify the format, numerical range, and semantic validity;

[0063] ③ Determine whether the key statements are consistent with the procedure template.

[0064] Thus, the detection results of the items that need to be verified in the first layer are obtained, and the decision function is defined as:

[0065]

[0066] in, It is a rule-based detection function that validates text based on a predefined set of rules. It is a set of rules such as regular expressions, field format rules, required field lists, and numerical ranges.

[0067] Second layer: Knowledge graph retrieval layer

[0068] The semantic vector obtained in step one The corresponding entity set is mapped to a knowledge graph, and structured verification is achieved through graph retrieval and relational reasoning.

[0069] For input entities The matching score between the graph node and the graph node is defined as follows:

[0070]

[0071] in,

[0072]

[0073] This represents all triples in the knowledge graph. Select the triplet that best matches the input. Indicates input The entity representation obtained by mapping, For entity semantic similarity function, To score the consistency of the relationship, These are the weighting coefficients.

[0074] Final judgment and The size relationship, among which This is a set threshold used to determine whether a match is sufficient. If If the current work order entry is found to be inconsistent with the procedural facts in the knowledge graph, then proceed to the third level of judgment.

[0075] Third layer: Large model semantic decision layer

[0076] The suspected problem items selected from the first two layers are compared with the relevant nodes and relationship chains obtained from the knowledge graph retrieval. Both are input into a Large Language Model (LLM) for consistency determination of contextual logic and security rules:

[0077]

[0078] in, It is a large language model (LLM) used to perform inference, decision-making, and result generation. Indicates the first The multimodal fusion vector of the terms serves as the semantic representation of the LLM input. Relevant subgraphs or relational chains retrieved from the knowledge graph to address this suspected issue are used as factual context in the LLM. Contextual constraints include on-site operating conditions, equipment topology, and operating mode.

[0079] Large models perform semantic reasoning based on prompt templates, for example:

[0080] "Please determine whether the following work order items are consistent with the procedure diagram, and point out potential safety hazards and suggestions for modification."

[0081] The final verification score is derived by combining the results from all three layers:

[0082]

[0083] in, These are multi-layered fusion weights, corresponding to the weights of the rule layer, graph layer, and large model layer, respectively. , , These represent the scores for the three levels respectively.

[0084] Step 4: In the output generation and feedback stage, the verification conclusions are returned in the form of a structured report, PDF export, or system interface, achieving real-time feedback and dynamic updates to the knowledge graph. Detailed steps are as follows:

[0085] The system outputs a structured intelligent verification report:

[0086]

[0087] in This indicates the field name or item identifier, such as the person in charge of the work or safety measures. This indicates the final score for this field. This refers to the system's explanation text and modification suggestions provided in natural language.

[0088] The intelligent verification report includes:

[0089] ① Field integrity and compliance test results;

[0090] ② The corresponding procedural clauses or safety measure nodes referenced in the knowledge graph;

[0091] ③ Suggested modifications and risk warnings.

[0092] The final report can be generated in PDF or JSON format and can be pushed to the power safety management system to achieve real-time feedback and dynamic updates of the knowledge graph.

[0093] In summary, the method of this invention not only improves the accuracy and intelligence of work order verification, but also realizes knowledge-driven logical reasoning and interpretable decision-making, providing efficient, intelligent and reliable new technical support for the safe production of power systems.

Claims

1. A method for intelligent verification of work tickets based on multimodal large-scale model knowledge graphs and large-scale model reasoning, characterized in that, The steps are as follows: Step 1: Multimodal Input Parsing: The system receives scanned PDF files and on-site images, extracts text feature vectors and image feature vectors through the OCR module and image encoder respectively, and forms a comprehensive semantic representation after multimodal fusion; Step 2: Knowledge Graph Construction: The system extracts entities and relationships from power safety regulations and historical operation data to generate a power safety knowledge graph, which is then stored and indexed in a graph database to provide structured knowledge support for subsequent reasoning. Define a knowledge graph as a set of triples: in: : A collection of entities, including equipment, personnel, measures, procedures and clauses, and operation tickets; : A set of relations, including "contains in", "applies to", "causes", and "constrains". : This represents a fact triple that states "there is a relation r between the head entity h and the tail entity t". Step 3, Layered Verification: Perform format and completeness checks on the basic fields of the ticket; calculate logical matching scores based on the graph relationship chain to identify potential security measure conflicts or omissions; use LLM to perform semantic reasoning in combination with context and graph nodes to generate semantic judgment results; the system finally integrates the three-layer results, calculates the comprehensive verification score, and outputs the intelligent judgment result. Step three employs a three-tiered verification mechanism, analyzing from simple to complex levels, specifically as follows: First layer: Rule matching layer Based on regular expressions and predefined rule sets Basic validation of key fields on the ticket, including: ① Check if any fields are missing; ② Verify the format, numerical range, and semantic validity; ③ Determine whether the key statements are consistent with the procedure template; Thus, the detection results of the items that need to be verified in the first layer are obtained, and the decision function is defined as: in, It is a rule-based detection function that validates text based on a predefined set of rules. It is a set of rules; Second layer: Knowledge graph retrieval layer The semantic vector obtained in step one The corresponding entity set is mapped to the knowledge graph, and structured verification is achieved through graph retrieval and relational reasoning; For input entities The matching score between the graph node and the graph node is defined as follows: in, This represents all triples in the knowledge graph. Select the triplet that best matches the input. Indicates input The entity representation obtained by mapping, For entity semantic similarity function, To score the consistency of the relationship, These are weighting coefficients; Final judgment and The size relationship, among which This is a set threshold used to determine whether a match is sufficient; if The current work order entry is considered inconsistent with the procedural facts in the knowledge graph, and the process proceeds to the third level of judgment. Third layer: Large model semantic decision layer The suspected problem items selected from the first two layers are compared with the relevant nodes and relationship chains obtained from the knowledge graph retrieval. Input them together into a large language model and perform a consistency check between the context logic and security rules: in, It is a large language model (LLM) used to perform inference, decision-making, and result generation. Indicates the first The multimodal fusion vector of the terms serves as the semantic representation of the LLM input. Relevant subgraphs or relational chains retrieved from the knowledge graph to address this suspected issue are used as factual context in the LLM. Contextual constraints include on-site operating conditions, equipment topology, and operating mode; The final verification score is derived by combining the results from all three layers: in, These are multi-layered fusion weights, corresponding to the weights of the rule layer, graph layer, and large model layer, respectively. , , These represent the scores for the three levels respectively; Step 4: Output Generation and Feedback: The verification conclusions are returned in the form of a structured report, PDF export, or system interface, enabling real-time feedback and dynamic updates of the knowledge graph.

2. The intelligent verification method for work tickets based on multimodal large-scale model knowledge graph and large-scale model reasoning as described in claim 1, characterized in that, The detailed steps for Step One are as follows: ① Perform OCR recognition on the input scanned PDFs and images to extract text content and table structure, resulting in a set of text paragraphs: in, It is the collection of text paragraphs or fields obtained after OCR parses the entire document. It is a set The first in A text paragraph or field, It is the total number of text paragraphs or fields, i.e., the set. The number of elements in the middle; ② Embed the image region into the image encoder to generate visual feature vectors: in, It is the first Image feature vectors of a single image or image region These are image coding models (such as CLIP, ViT, etc.) used to map images to vector space; Indicates the first Zhang input image or image region; ③ Encode text paragraphs into semantic vectors: in, It is related to the text segment The corresponding text semantic vector is the text encoding result; This represents a text encoding model used to map text to a vector space. ④ Use a multimodal fusion model to obtain a comprehensive feature representation: in, Indicates the first Each entry is a multimodal fusion semantic representation vector that integrates text and image information. The fusion weights are used to balance the contributions of text and image features.

3. The intelligent verification method for work tickets based on multimodal large-scale model knowledge graphs and large-scale model reasoning as described in claim 2, characterized in that, Step two of the construction process includes: Entity extraction: Identifying key objects in the invoice text using named entity recognition algorithms; Relation extraction: Extracting logical relationships between entities based on dependency parsing and template rules; Attribute fusion and deduplication: Align entities with the same name or synonyms to build a unified entity library; Graph structure storage: Utilizes a graph database to store graph nodes and edges, and supports query and reasoning operations; Knowledge graphs support calculating semantic similarity through relational chains: in This indicates the path length considered during matching. Indicates the index of the relationship chain. For relation matching functions, targeting the first... Relationships contained in the path If the relationship matches the expectation, a higher value will be obtained; This is the path weight, used to measure the importance of the path.

4. The intelligent verification method for work tickets based on multimodal large-scale model knowledge graph and large-scale model reasoning as described in claim 3, characterized in that, Step four is detailed below: The system outputs a structured intelligent verification report: in This indicates the field name or item identifier, such as the person in charge of the work or safety measures. This indicates the final score for this field. This refers to the system's explanation text and modification suggestions provided in natural language. The intelligent verification report includes: ① Field integrity and compliance test results; ② The corresponding procedural clauses or safety measure nodes referenced in the knowledge graph; ③ Suggested modifications and risk warnings; The final report can be generated in PDF or JSON format and pushed to the power safety management system to achieve real-time feedback and dynamic updates of the knowledge graph.

Images