Power entity joint relation extraction method and system based on multi-modal large model

Abstract

The invention relates to an electric power entity joint relation extraction method and system based on a multi-modal large model in the technical field of electric power system automation control, and the method is used for inputting collected electric power texts and electric power equipment images into a trained electric power entity joint relation extraction model. The entity-relation triple and a visual knowledge graph are output; the power entity joint relation extraction model comprises a data acquisition module; a text preprocessing module; a text encoder (BERT-Base); a visual encoder (Swin Transform) is arranged; a fusion layer; a cascading type binary annotation framework CasRel is adopted; according to the method, through multi-modal fusion and a joint extraction framework, the technical problems that an existing electric power entity relation extraction method is difficult to identify an overlapping relation in an electric power text, the multi-modal data fusion efficiency is low, the field adaptability is poor and the like are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system automatic control, in particular to a power entity joint relationship extraction method based on a multi-modal large model and a power entity joint relationship extraction system based on a multi-modal large model. BACKGROUND

[0002] The current power industry has a wide range of data sources, including text, images, sensor monitoring data and other modalities. Traditional power field knowledge graph construction methods usually use rule matching or independent model to process each modality data, and then integrate it manually, resulting in low processing efficiency and easy errors, and low multi-modal data fusion efficiency. If the problem of low multi-modal data fusion efficiency is not solved, it will lead to long construction period and high cost of power knowledge graph, which is difficult to meet the demand of real-time analysis and decision support of power system. Traditional methods need to rely on artificial rules or independent models to process each modality data, and then integrate it through complex processes, which not only consumes time and effort, but also easily produces information islands and data redundancy.

[0003] There are a large number of single entity overlap relationships (SEO) and entity pair overlap relationships (EPO) in power text. The traditional pipeline method (first entity recognition and then relationship extraction) has the problem of backward transmission of error information, which leads to a significant decrease in the accuracy of overlap relationship recognition, and the accuracy of overlap relationship recognition is insufficient. If the problem of insufficient overlap relationship recognition accuracy is not solved, it will lead to incomplete entity relationships in the power knowledge graph, affecting the accuracy of subsequent applications (including fault diagnosis, equipment maintenance, etc.). SEO and EPO relationships are common in power text, for example, in sentences such as "substation A connects line B and line C, and line B fault causes substation A and line C power interruption", substation A participates in two relationships at the same time, and the traditional pipeline method is prone to omission or incorrect identification.

[0004] When a general large model is applied in the power field, it needs a large amount of labeled data for fine-tuning due to the lack of power professional knowledge, and its performance is limited. In addition, power field data has the characteristics of strong professionalism and rich terminology, and general models are difficult to effectively capture these characteristics, and have poor domain adaptability. If the problem of poor domain adaptability is not solved, it will lead to the dependence of power knowledge graph construction on a large amount of manual annotation, increasing cost and reducing efficiency. When a general large model is applied in the power field, it is difficult to accurately identify professional terminology and understand complex business logic, resulting in a decrease in the accuracy of entity recognition and relationship extraction, and failing to meet the high-quality requirements of the power industry for knowledge graph. The operating state and equipment information of the power system are constantly changing, and the knowledge graph needs to be updated in real time to reflect the latest situation. Traditional methods use periodic batch updates, which cannot meet the real-time requirements, and the dynamic update mechanism is imperfect. SUMMARY

[0005] (1) Technical problems to be solved In order to solve the technical problems that the existing power entity joint relationship extraction method is difficult to identify overlapping relationships in power text, the multi-modal data fusion efficiency is low, and the domain adaptability is poor, the present application provides a power entity joint relationship extraction method and system based on a multi-modal large model.

[0006] (2) Technical scheme In a first aspect, the present application provides a power entity joint relationship extraction method based on a multi-modal large model, which is used to input the collected power text and power equipment images into a trained power entity joint relationship extraction model, and can output entity-relation triples and viewable knowledge graphs; the power entity joint relationship extraction model comprises: A data acquisition module is used to acquire power text and power equipment images in a target area; A text preprocessing module is used to process the collected power text into preprocessed power text; A text encoder BERT-Base is used to process the preprocessed power text into a text semantic feature vector; A visual encoder Swin Transformer is used to process the power equipment images into an image semantic feature vector; A fusion layer is used to fuse the text semantic feature vector and the image semantic feature vector to form a fusion feature vector; A cascaded binary labeling framework CasRel is used to extract entity-relation triples from the fusion feature vector and generate a knowledge graph; A conflict detection module is used to determine whether the extracted entity-relation triples conflict with existing relationships, if yes, a conflict resolution mechanism is triggered, otherwise, the detected entity-relation triples are output.

[0007] As a further improvement of the above-mentioned scheme, the power text preprocessing method comprises: Word segmentation: based on the ontology dictionary for word segmentation to solve the problem of professional term recognition; Part-of-speech tagging: define a part-of-speech tagging system according to the characteristics of the power field; Remove stop words: identify and remove irrelevant words based on the ontology dictionary; Feature selection: use the WTF-IDF optimization algorithm to improve the accuracy of feature selection by weighting power field professional vocabulary.

[0008] As a further improvement of the above-mentioned scheme, the ontology dictionary is a power field ontology dictionary, which includes power field professional vocabulary; the dictionary construction method comprises: Collecting defects, defect elimination, maintenance, testing, and other records or reports of various power equipment; Organizing guidelines and standards related to power equipment issued by enterprises; Collect various work orders, work tickets, operation tickets, etc. of power enterprises; Combined with existing external general dictionaries, a statistical-based word segmentation model is used to segment the corpus, and the word segmentation results are sorted based on word frequency; Relies on personnel with professional knowledge in the power field to correct professional terms, colloquial language, noun-piled word strings and phrase vocabularies, etc. to form the final power ontology dictionary.

[0009] As a further improvement of the above scheme, the text encoder is based on the BERT architecture and is pre-trained on power professional literature, equipment manuals, fault reports and other data for field adaptation.

[0010] As a further improvement of the above scheme, the visual encoder uses a lightweight visual Transformer to extract image features, including device appearance, location, and connection relationship.

[0011] As a further improvement of the above scheme, the structure of the fusion layer includes: S1, alignment mechanism: use a learnable mapping matrix And , map the text features And visual features To a unified semantic space: ; S2, cross attention: calculate the interaction between text and visual features to generate fusion features; Calculate the attention weight of the text to the vision And the attention weight of the vision to the text : ; Where Q and K are query and key vectors generated by linear transformation; The fused features are: ; S3, weighted sum: dynamically adjust the weight according to the importance of the modal features α , β , form the final fusion features ; ; As a further improvement of the above scheme, the cascaded binary labeling framework CasRel includes: S1, entity recognition: apply the cascaded binary labeling framework to the fusion features ; Head entity recognition: Using a linear layer and sigmoid activation function, determine if each token is the start / end position of a head entity; Tail entity recognition: Based on the head entities, identify tail entities; generate a candidate head entity set. ; S2, Joint Identification of Relationships and Tail Entities: For each The dual affine attention mechanism is used to calculate its relationship with all candidate tail entities. Relationship probability: ; in For relational type parameter matrix, The sigmoid function is used to select the relation type with the highest probability and its corresponding tail entity to form a triple. ; S3. Knowledge Graph Construction: Constructing Triples Stored in the Neo4j graph database to form a knowledge graph.

[0012] As a further improvement to the above scheme, the dual affine attention mechanism is as follows: Head entity Tail-end entity Features Perform affine transformation: ; Constructing the feature interaction matrix : ; Extracting relational features using max pooling: ; As a further improvement to the above scheme, the conflict detection module uses SHACL and TBox content to perform consistency checks on newly extracted relationships to identify whether there are conflicting relationships; when a conflict is detected between a newly extracted relationship and existing knowledge, the conflict resolution mechanism is triggered. and, The conflict resolution mechanism adopts a hierarchical decision-making strategy: it prioritizes automatic correction, and if automatic correction fails or the risk of conflict is high, it triggers an expert review mechanism.

[0013] Secondly, the present invention also provides a power entity joint relation extraction system based on a multimodal large model, which adopts the power entity joint relation extraction method based on a multimodal large model as described in the first aspect. By inputting the collected power text and power equipment images into the trained power entity joint relation extraction model, entity-relationship triples and a visualized knowledge graph are obtained.

[0014] (3) Beneficial effects 1. Significantly improved accuracy in relation extraction This invention achieves an F1 score of 93.2% in power text relationship extraction tasks through multimodal fusion and joint extraction framework, which is approximately 9.4% higher than the traditional method Bi-LSTM+CRF. In particular, when dealing with overlapping relationships between SEO and EPO, the accuracy is improved from 83.76% to 93.2% of the traditional method, effectively solving the problem of insufficient accuracy in recognizing overlapping relationships in complex power texts by traditional methods.

[0015] 2. Improved efficiency of multimodal data fusion This invention employs a lightweight visual Transformer and a cross-attention mechanism, reducing the computational resource consumption for multimodal data fusion by approximately 40% and increasing processing speed by approximately 2.5 times. When processing a mixed dataset containing 1000 images of power equipment and 100,000 pieces of power-related text, traditional methods require approximately 8 hours, while this invention only requires approximately 3 hours, significantly improving the efficiency of multimodal data fusion.

[0016] 3. Enhanced domain adaptability This invention improves the model's adaptability in the power industry by approximately 15-20% through domain pre-training and rule constraints. The general BERT model achieves approximately 73.4% accuracy in entity recognition within the power industry, while the pre-trained language model of this invention achieves 93.2% accuracy. Similarly, the general model achieves approximately 74.4% accuracy in relation recognition within the power industry, while the model of this invention achieves 94.5% accuracy. 4. Knowledge graph update latency has been significantly reduced. This invention reduces the knowledge graph update latency from 72 hours using traditional methods to less than 30 seconds through online learning and incremental update mechanisms. For example, when handling power grid fault events, traditional methods require manual review and batch updates, a process that can take hours or even days. In contrast, this invention, through automatic detection, extraction, and fusion, reduces the entire process from data collection to graph update to less than 1 minute, effectively solving the problem of imperfect dynamic update mechanisms for knowledge graphs.

[0017] 5. Significantly reduced labeling costs This invention reduces reliance on manual annotation through a multimodal fusion and joint extraction framework. For example, in power equipment fault diagnosis, the time required for a single diagnosis is reduced from 20 working days using traditional methods to 5 working days. It is estimated that this can save 1500 person-days of diagnostic work annually across the province, resulting in annual cost savings of approximately 450,000 yuan based on an average daily wage of 300 yuan. In power dispatch strategy generation, the time for a single power transfer decision is reduced from 30 minutes to 1 minute, and the time for developing large-scale power supply guarantee plans is reduced from 10 hours to 10 minutes, significantly lowering annotation costs and manual workload. Attached Figure Description

[0018] Figure 1 This is a module diagram of the power entity joint relationship extraction model.

[0019] Figure 2 It is a workflow diagram of the collaborative process between the expert review mechanism and the automatic correction strategy. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that when a component is said to be "installed on" another component, it can be directly on the other component or it may be in a component that is centered on it. When a component is said to be "set on" another component, it can be directly set on the other component or it may also be in a component that is centered on it. When a component is said to be "fixed to" another component, it can be directly fixed to the other component or it may also be in a component that is centered on it.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] This invention provides a method for extracting joint relationships between power entities based on a multimodal large model. The operation steps of the joint relationship extraction method are as follows: Collected power text and images of power equipment are input into a trained power entity joint relationship extraction model, which outputs entity-relationship triples and a viewable knowledge graph. This relationship extraction method can efficiently extract the associations between entities in the power field from multi-source heterogeneous data, solving problems such as difficulty in identifying overlapping relationships in power text, low efficiency of multimodal data fusion, and poor domain adaptability. Please refer to... Figure 1 , Figure 1 A module diagram for extracting the association relationship model of power entities, the model including: 1. Data acquisition module, used to acquire power text and images of power equipment within the target area.

[0024] II. Text preprocessing module, used to process the collected power text into preprocessed power text.

[0025] 3. The text encoder BERT-Base is used to transform preprocessed electricity text into text semantic feature vectors.

[0026] IV. The visual encoder, Swing Transformer, is used to transform images of power equipment into image semantic feature vectors.

[0027] Fifth, the fusion layer is used to fuse text semantic feature vectors and image semantic feature vectors to form a fused feature vector.

[0028] VI. CasRel, a cascaded binary annotation framework, is used to extract entity-relation triples from fused feature vectors and generate knowledge graphs.

[0029] 7. Conflict detection module, used to determine whether the extracted entity-relationship conflicts with existing relations. If so, the conflict resolution mechanism is triggered; otherwise, the entity-relationship triple is output.

[0030] This invention significantly improves the accuracy of power entity relationship extraction, significantly increases the efficiency of multimodal data fusion, greatly reduces the knowledge graph update speed delay, and significantly reduces annotation costs. At the same time, it solves the technical problems of existing power entity relationship extraction methods, such as difficulty in identifying overlapping relationships in power text, low efficiency of multimodal data fusion, and poor domain adaptability.

[0031] The following sections will introduce each module in detail.

[0032] I. Data Acquisition Module.

[0033] The system acquires power-related text and images of power equipment within the target area. The power-related text is then input into the text preprocessing module, and the images of power equipment are input into the visual encoder, Swing Transformer.

[0034] II. Text Preprocessing Module.

[0035] The collected power data is standardized and preprocessed to obtain preprocessed power data. The preprocessing methods include: (1) Word segmentation: Word segmentation is performed based on the ontology dictionary to solve the problem of professional terminology recognition; (2) Part-of-speech tagging: A custom part-of-speech tagging system is defined according to the characteristics of the power industry, such as "large components", "small components", "attributes", "degrees", etc.; (3) Stop word removal: Identify and remove irrelevant words based on the ontology dictionary; (4) Feature selection: The WTF-IDF optimization algorithm is adopted to improve the accuracy of feature selection by weighting professional terms in the power field.

[0036] In this embodiment, the ontology dictionary contains approximately 3,000 professional terms in the power industry; the text segmentation accuracy is improved to 97.5%; the part-of-speech tagging coverage is improved to 96.2%; the stop word removal rate is improved to 99.3%; and the α (weighted value of feature terms) in the ETF-IDF algorithm is set to 1.6.

[0037] The word segmentation method uses an ontology dictionary specific to the power industry, which includes specialized terms such as equipment terminology, fault types, and operating procedures, including terms like "interconnected users," "high-voltage fuse," and "meter verification." The methods for constructing this power industry ontology dictionary include: (1) Collect records or reports on defects, troubleshooting, maintenance, and testing of various power equipment; (2) Compile the guidelines and standards related to power equipment issued by the company; (3) Collect various work orders, work permits, operation permits, etc. from power companies; (4) Combining existing external general dictionaries, a statistical-based word segmentation model is used to segment the corpus, and the segmentation results are sorted based on word frequency; (5) Relying on personnel with expertise in the power field to revise professional terms, colloquialisms, word strings and phrases, etc., to form the final power dictionary.

[0038] III. Text Encoder BERT-Base.

[0039] A pre-trained language model for the power industry is used to encode text and extract semantic features, transforming pre-processed power industry text into semantic feature vectors. This model, based on the BERT architecture, is pre-trained on power industry literature, equipment manuals, and fault reports to better understand power industry terminology and semantic relationships. In this embodiment, the text encoder is based on the BERT-Base architecture with approximately 110 million parameters. Building upon the general BERT model, domain-adaptive pre-training using power industry literature, equipment manuals, and fault reports enhances the model's ability to understand power industry terminology and semantics.

[0040] IV. Visual Encoder: Swing Transformer.

[0041] A lightweight visual Transformer is employed to extract image features, transforming power equipment images into semantic feature vectors, including equipment appearance, location, and connectivity. A hierarchical window attention mechanism efficiently extracts local and global features from power equipment images, reducing computational resource consumption by approximately 40% compared to traditional Transformers. In this embodiment, the visual encoder uses the Swing Transformer, Base version, with approximately 89 million parameters.

[0042] V. Integration Layer.

[0043] A multimodal fusion module is designed to fuse textual and visual features, specifically by combining textual semantic feature vectors and image semantic feature vectors to form a fused feature vector. Specific implementation methods include: (1) Alignment mechanism: The features of different modalities are mapped to the same semantic space through a learnable mapping matrix.

[0044] (2) Cross-attention mechanism: Design a multi-head attention module to capture the interaction between text and visual features.

[0045] (3) Fusion strategy: adopt a weighted summation method and dynamically adjust the weights according to the importance of features.

[0046] In this embodiment, the fusion layer includes a cross-attention mechanism with six attention heads, and the fusion strategy includes: 1) Alignment mechanism: The text and visual features are mapped to the same semantic space through a learnable mapping matrix.

[0047] Feature alignment: using a learnable mapping matrix and Text features and visual features Mapping to a unified semantic space: ; 2) Cross-attention: Calculate the interaction between text and visual features to generate fused features.

[0048] Cross-attention mechanism: Calculate the attention weights of text to vision. visual attention weights to text : ; Where Q and K are query and key vectors, generated through linear transformation; The characteristics after fusion are: ; 3) Weighted summation: The weights are dynamically adjusted based on the importance of the modal features. α , β This forms the final fusion characteristic; ; The weights are automatically optimized using learnable parameters or task objectives (such as relation extraction accuracy).

[0049] VI. Cascaded binary annotation framework CASREL.

[0050] The joint relation extraction employs an improved CasRel framework, combined with a dual affine attention mechanism, to directly extract entity-relation triples from multimodal fusion features: Entity recognition: First, identify all possible head entities in the text. A cascaded binary labeling framework is used to determine whether each token is the start or end position of a head entity through a linear layer and a sigmoid activation function. The nearest matching principle is used to pair start and end to obtain a set of candidate head entities.

[0051] Joint Relationship and Tail Entity Identification: For each identified head entity, possible relationship types and corresponding tail entities are identified. The improved CasRel framework constructs a feature interaction matrix between head and tail entities through a dual affine attention mechanism, capturing complex semantic relationships between entities.

[0052] Knowledge graph construction: The extracted entity-relation triples are stored in a graph database, and knowledge representation methods such as RDF (Resource Description Framework) or Property Graph are used to construct a knowledge graph for the power industry.

[0053] The specific steps include: (1) Entity alignment: solves the problem of multiple representations of the same entity in different modal data; (2) Relation disambiguation: handling ambiguous and polysemous relations; (3) Graph optimization: The knowledge graph is completed and its quality is checked based on graph neural network (GNN).

[0054] In this embodiment, the entity recognition accuracy is 95.8%; the relation recognition accuracy is 94.5%; the triplet extraction F1 value is 93.2%; the knowledge graph is stored in the Neo4j graph database; the number of graph nodes is 320,000+; the number of graph relations is 1.28 million+; and the update latency is <30 seconds.

[0055] (1) Entity recognition: 1) Fusion features Applying a cascaded binary annotation framework: Head entity recognition: Using a linear layer and sigmoid activation function, determine whether each token is the start / end position of the head entity.

[0056] Tail entity recognition: Based on the head entities, tail entities are recognized using a similar method.

[0057] 2) Generate a set of candidate head entities .

[0058] (2) Joint identification of relations and tail entities: 1) For each The dual affine attention mechanism is used to calculate its relationship with all candidate tail entities. Relationship probability: ; in For relational type parameter matrix, This is the sigmoid function.

[0059] 2) Select the relation type with the highest probability and its corresponding tail entity to form a triple. .

[0060] (3) Knowledge graph construction: The triples are stored in the Neo4j graph database, and the graph quality is optimized through entity alignment, relation disambiguation, and GNN completion.

[0061] This embodiment employs an improved CasRel framework, combined with a dual affine attention mechanism and a joint extraction strategy. In SEO / EPO overlapping relationship scenarios, the dual affine mechanism can more accurately model complex dependencies between entities, improving accuracy to 93.2%, a 9.4% improvement over traditional methods. The dual affine attention mechanism is as follows: Head entity Tail-end entity Features Perform affine transformation: ; Constructing the feature interaction matrix : ; Extracting relational features using max pooling: ; The joint extraction strategy is as follows: directly from Predict the relationship type to avoid error propagation in pipelined methods.

[0062] The interaction strength between the head entity and the tail entity is controlled by gating mechanisms (such as sigmoid).

[0063] VII. Conflict Detection Module.

[0064] In the dynamic updating of knowledge graphs in the power sector, conflict detection is a crucial step in ensuring the consistency of the knowledge graph. In this embodiment, SHACL (Shape Constraint Language) and TBox content are used to perform consistency checks on newly extracted relationships and identify potential conflicting relationships. When a conflict is detected between a newly extracted relationship and existing knowledge (e.g., "Substation A connects to line B" contradicts "Line B disconnects from substation A"), the system must trigger a conflict resolution mechanism. This mechanism employs a hierarchical decision-making strategy: it prioritizes automatic correction; if automatic correction fails or the conflict risk is high, an expert review mechanism is triggered.

[0065] 1. Automatic correction strategy: Intelligent closed-loop repair of low-risk conflicts.

[0066] (1) Triggering conditions: Low conflict risk level: Conflicts are common patterns (such as equipment status updates, redundancy relationships), and the confidence level is >90%.

[0067] High success rate of automatic correction: Conflict types can be covered by predefined rules (e.g., "line disconnection" is automatically associated with "power interruption").

[0068] Typical power scenario example: New relationship: "Line C failed at 14:05 on 2023-10-05"; Existing relationship: "Line C was running at 14:00 on 2023-10-05" (older timestamp); Conflict type: Device status update (low risk, routine maintenance).

[0069] (2) Technical implementation of the automatic correction strategy: 1) Rule-based correction engine (core innovation): Rule source: A dynamic rule base built based on the power domain ontology dictionary (Technical Disclosure Document "Power Domain Rule Base"), including: Business rule: IF line fault THEEN power interruption AND trigger alarm (derived from "Guidelines for the Safety and Stability of Power Systems"); Time rule: IF new relationship time > existing relationship time THEN update status; Equipment rules: Transformer connection lines (otherwise a conflict will be triggered); Correction logic: if conflict_type == "state update" and new_time > existing_time: auto_correct = "update existing relationship timestamp + add new state" elif conflict_type == "logical redundancy" and rule_exists("line fault → power interruption"): auto_correct = "automatically add missing relationship (power interruption)" else: trigger_expert_audit() Low confidence time artificial 2) Multimodal evidence fusion correction (an extension of the "Multimodal Feature Fusion" section in the technical disclosure): When a conflict involves a contradiction between text and an image (e.g., the text describes "Line C is faulty," while the image displays "Line C is intact"), the system: Calculate image credibility by calling visual Transformer features (based on power equipment image recognition model).

[0070] Prioritize modalities with high credibility (e.g., if the image credibility is >85%, then the image shall be used).

[0071] Example: Image analysis shows "Line C insulator damaged" (high confidence), automatically correcting the text relationship to "Line C fault".

[0072] 3) Confidence level dynamic threshold management: Table 1: Example Table of Confidence Level Dynamic Threshold Management Conflict type Confidence threshold Automatic correction success rate Power field example Device status update 90% 98.2% Line operation → fault (timestamp update) Logical relationship redundancy 92% 95.7% Line fault → power supply interruption (rule coverage) Multi-modal evidence conflict 95% 89.1% Text "fault" vs. image "intact" High-risk safety conflict - 0% Transformer not connected line (trigger expert review) 4) Post-correction verification and rollback mechanism: After the correction, the system automatically reruns the SHACL check to ensure there are no new conflicts.

[0073] If the correction fails (e.g., a new conflict arises), the system will automatically roll back to the state before the correction and trigger an expert review (to prevent the accumulation of errors).

[0074] Electricity value: In equipment inspection scenarios, automatic correction reduces the "status update delay" from 2 hours to within 10 seconds (the technical disclosure states "incremental update delay of 10-30 seconds").

[0075] (3) Power sector compatibility description: To avoid the shortcomings of general models: General large models cannot understand the logic of "line fault → power outage". This strategy achieves domain adaptation through power rule base, and the success rate of correction is 32.5% higher than that of general methods.

[0076] Patent innovation points: Transform power industry regulations into enforceable rules (e.g., transform DL / T 572 into TBox constraints).

[0077] Design a dynamic confidence threshold to balance automation and security (automatic repair for low-risk cases, manual repair for high-risk cases).

[0078] Example value: In power grid dispatching, the automatic correction of the "line fault → power outage" relationship reduces the fault handling time from 15 minutes to 2 minutes, improving response efficiency.

[0079] 2. Triggering expert review mechanism: a closed-loop handling process for high-risk conflicts.

[0080] (1) Triggering conditions: High conflict risk level: The system classifies conflicts into risk levels based on SHACL / TBox detection results (based on power sector safety thresholds): High-risk conflicts: involving the status of core equipment (such as transformers and circuit breakers), safety rules (such as "line disconnection trigger alarm"), or violations of mandatory standards in the power industry (such as DL / T 572-2013 "Operating Procedures for Power Transformers").

[0081] Confidence threshold: Conflict confidence > 95% (supporting indicator of "conflict detection accuracy of 98.3%" in the technical disclosure), and the automatic correction strategy cannot cover it (such as conflict involving multimodal evidence contradictions).

[0082] Typical power scenario example: New information: "The main transformer Y1 of 500kV substation A will be shut down at 14:00 on October 5, 2023." Current information: "The main transformer Y1 of 500kV substation A was running at 14:00 on October 5, 2023" (from real-time data of the SCADA system). Conflict type: Equipment status conflict (high risk, directly related to power grid security).

[0083] (2) Implementation process of the expert review mechanism (patent protectable points): 1) Automatic Conflict Capture and Classification (System Side): The SHACL engine performs TBox constraint checks (such as "Device status matches timestamp") and generates a conflict report.

[0084] The report includes: Conflicting relationship pairs (new relationship vs. existing relationship); Types of conflict (factual conflict / logical conflict); Chain of evidence (text source, image evidence, timestamp, confidence level); Power risk level label (e.g., "Level 1 Safety Risk": involving main equipment outage); (Technical parameters: Conflict detection accuracy 98.3%, risk classification accuracy 96.5%).

[0085] 2) Expert review work order push (system side): The system automatically pushes conflict reports to the expert review platform of the power dispatch center (integrated into the knowledge graph system), triggering real-time alarms (such as WeChat / SMS).

[0086] The work order includes: 3D visualization of conflict views (comparison of map nodes, timeline, and overlay of multimodal evidence); Suggested decision options (e.g., "Confirm shutdown", "Correct timestamp", "Ignore"); Historical conflict reference (records of similar conflict resolution, based on the power industry knowledge base).

[0087] 3) Expert decision-making closed loop (human terminal): Expert role: Power system operation expert (must have dispatcher qualifications, such as State Grid certified "senior dispatcher").

[0088] The review process includes verifying evidence: retrieving real-time data from the SCADA system and maintenance work orders (such as "Main Transformer Y1 Maintenance Application Form") to confirm the authenticity of the conflict.

[0089] Decision types include: Correction Confirmation: If the new relationship is confirmed to be correct ("Main transformer Y1 is actually out of service"), then update the knowledge graph and mark it as "Expert Correction".

[0090] Ignore false alarms: If the conflict originates from a data acquisition error (such as a sensor malfunction), ignore the new relationship and mark it as a "false alarm".

[0091] Escalation process: If the conflict involves cross-regional power grids (such as the contradiction between "substation A connects to line B" and "line B belongs to substation C"), it will be automatically escalated to the provincial dispatch center.

[0092] Feedback loop: Expert decisions are fed back to the system in real time to optimize the automatic correction rule base (such as adding a rule that "main transformer outage requires associated maintenance work order").

[0093] (3) Power sector compatibility description: Traditional methods (such as manual review) take several hours, while this mechanism reduces the review time to 5-15 minutes through system-driven order push (the technical disclosure states "adaptive optimization response time < 5 seconds").

[0094] Patent innovation: The power industry safety regulations (such as the "Regulations on Emergency Response and Investigation of Power Safety Accidents") are internalized as TBox constraints, so that the conflict risk classification conforms to the mandatory industry standards and avoids the "safety blind spots" of general knowledge graphs.

[0095] Example value: In power grid fault diagnosis, it avoids misjudgments caused by equipment status conflicts (such as misjudging "fault outage" as "normal operation"), and reduces the risk of accident escalation.

[0096] 3. The collaborative working logic of the expert review mechanism and the automatic correction strategy Table 2: Example of the collaborative working logic between the expert review mechanism and the automatic correction strategy Link Automatic correction strategy Trigger expert review mechanism Trigger condition Low-risk conflict (confidence > 90%) High-risk conflict (confidence > 95%) or automatic failure Response speed <5 seconds (automatically performed at system level) 5-15 minutes (expert review) Correction success rate 95.7% (common scenarios in power) 100% (manual decision) Value in the field of electricity Eliminate 90% of routine conflicts and improve update efficiency Ensure zero misjudgment for safety core conflicts and comply with industry regulations Core Dynamic construction of rule base, management of confidence threshold Expert review work order is deeply bound with power safety regulations Please see Figure 2 , Figure 2 This is a flowchart illustrating the collaborative workflow of the expert review mechanism and the automatic correction strategy. When the power entity relationship extraction model identifies a conflict in the extracted relationships, it triggers a conflict resolution mechanism, which involves the collaborative operation of the expert review mechanism and the automatic correction strategy. First, it determines whether the conflict is high-level or low-level. If it's low-level, the automatic correction strategy is activated to correct the conflict. If the correction is successful, the knowledge graph is updated; otherwise, expert review is triggered. Expert review is also triggered when the conflict level is high-level. Simultaneously, the conflict is automatically captured and a conflict report is generated. The conflict report is uploaded to the expert review platform, triggering a real-time alarm to alert power system operation experts. Power system operation experts make decisions based on the conflict report: if correction is made, the knowledge graph is updated; if ignored or escalated, it is recorded in the knowledge base.

[0097] In this embodiment, the conflict detection accuracy is 98.3%; the incremental update latency is 10-30 seconds; the online learning convergence speed is 5 times faster than traditional offline learning; the adaptive optimization response time is <5 seconds; and the knowledge graph update frequency is real-time (<1 minute).

[0098] The working principle of the power entity joint relationship extraction method is based on multimodal representation learning and joint extraction framework, realizing relationship extraction in the construction of knowledge graph in the power field.

[0099] The multimodal large model architecture adopts a three-stage architecture of "encoding-fusion-decoding". First, the text and visual data are encoded separately. Then, the multimodal features are fused through a cross-attention mechanism. Finally, the relationship is extracted based on the fused features.

[0100] Text encoding: Encode text using a pre-trained model in the power industry to capture power industry terminology and semantic relationships.

[0101] Visual encoding: Lightweight visual Transformers (such as Swing Transformer) are used to extract image features, including device appearance, location, and connectivity.

[0102] Feature fusion: A multimodal fusion module is designed to map features from different modalities to the same semantic space through a learnable mapping matrix, and to capture the interaction relationships between modalities through a cross-attention mechanism.

[0103] Relation extraction: The improved CasRel framework, combined with a dual affine attention mechanism, directly extracts entity-relation triples from the fused features.

[0104] The power sector knowledge injection mechanism injects power sector knowledge into the model through domain pre-training, rule constraints, and knowledge embedding. Specifically: Domain pre-training: Based on the general BERT, domain-adaptive pre-training is performed using data such as power industry literature, equipment manuals, and fault reports to enhance the model's ability to understand power industry terminology and semantics.

[0105] Rule constraints: Design a rule base for the power industry (including "transformer connection lines") and impose constraints during model training and inference to ensure the rationality of extracted relationships.

[0106] Knowledge embedding: embedding terms from the power industry ontology dictionary into the model parameters to improve the accuracy of professional terminology recognition.

[0107] Dynamic reasoning optimization strategies include real-time updates, adaptive adjustments, and conflict resolution, among which: Real-time updates: New data is integrated into the knowledge graph in real time using online learning technology, with an update delay of less than 30 seconds. In this embodiment, new data is integrated into the knowledge graph in real time based on online learning technology. Specific implementation includes: (1) Incremental feature extraction: Only new data is processed to avoid repeated calculations; (2) Incremental relation extraction: Fine-tuning the existing model to improve update efficiency; (3) Incremental knowledge fusion: Graph neural networks are used for local updates to maintain the stability of the overall graph structure.

[0108] Adaptive Adjustment: Based on user query behavior and knowledge graph usage, the model parameters and extraction strategy are dynamically adjusted to improve system response speed and accuracy. In this embodiment, the model parameters and extraction strategy are dynamically adjusted based on user query behavior and knowledge graph usage. For example, when it is detected that users frequently query "reasons for device malfunctions," the extraction weight of related entity relationships is automatically increased.

[0109] Conflict resolution: The newly extracted relationships are checked for consistency using SHACL and TBox content. When a conflicting relationship is found, an expert review mechanism or an automatic correction strategy is triggered.

[0110] Based on the above embodiments, this invention also proposes a power entity joint relationship extraction system based on a multimodal large model, which uses the power entity joint relationship extraction method based on a multimodal large model.

[0111] In summary, the key innovations of this invention are as follows: 1. Cascaded Relationship Extraction through Multimodal Fusion The core innovation of this invention lies in the design of a multimodal fusion-based cascaded relation extraction method. This method fuses textual and visual features during the encoding stage to form a unified semantic representation, and then extracts relations based on the fused features. This approach avoids the shortcomings of traditional methods, such as independent processing of each modality and manual integration, thus improving the efficiency of multimodal data fusion.

[0112] 2. Joint extraction based on dual affine attention mechanism This invention employs an improved CASREL framework combined with a dual affine attention mechanism to directly extract entity-relationship triples from fused features, avoiding the error accumulation of pipelined methods. The dual affine attention mechanism captures complex semantic relationships between entities by constructing a feature interaction matrix between the head and tail entities, demonstrating particularly strong performance in handling overlapping SEO and EPO relationships.

[0113] 3. Knowledge-driven dynamic reasoning optimization This invention designs a dynamic reasoning strategy involving conflict detection, incremental updates, and adaptive optimization, enabling real-time updates and optimization of the knowledge graph. Newly extracted relationships are checked for consistency using SHACL and TBox content. When conflicting relationships are detected, an expert review mechanism or an automatic correction strategy is triggered to ensure the accuracy and consistency of the knowledge graph.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for extracting joint relationships of power entities based on a multimodal large model, characterized in that, The collected power text and images of power equipment are input into the trained power entity joint relationship extraction model, which outputs entity-relationship triples and a visualized knowledge graph. The power entity joint relationship extraction model includes: The data acquisition module is used to collect power-related text and images of power equipment within the target area; The text preprocessing module is used to process the collected power text into preprocessed power text; The text encoder BERT-Base is used to process preprocessed electricity text into text semantic feature vectors; The visual encoder Swin Transformer is used to process images of power equipment into image semantic feature vectors. The fusion layer is used to fuse text semantic feature vectors and image semantic feature vectors to form a fused feature vector. CasRel, a cascaded binary annotation framework, is used to extract entity-relation triples from fused feature vectors and generate knowledge graphs. The conflict detection module is used to determine whether the extracted entity-relation triples conflict with existing relations. If they do, the conflict resolution mechanism is triggered; otherwise, the detected entity-relation triples are output.

2. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 1, characterized in that, Methods for preprocessing electrical data include: Word segmentation: Word segmentation is performed based on an ontology dictionary to solve the problem of professional terminology recognition; Part-of-speech tagging: A customized part-of-speech tagging system based on the characteristics of the power industry; Stop word removal: Identify and remove irrelevant words based on an ontology dictionary; Feature selection: The WTF-IDF optimization algorithm is adopted to improve the accuracy of feature selection by weighting professional terms in the power industry.

3. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 2, characterized in that, The ontology dictionary is a dictionary for the power industry, containing specialized terms in the field. The dictionary construction methods include: Collect records or reports of defects, troubleshooting, maintenance, and testing of various power equipment; Organize and compile guidelines and standards related to power equipment issued by enterprises; Collect various work orders, work permits, and operation permits from power companies; By combining existing external general-purpose dictionaries, a statistical-based word segmentation model is used to segment the corpus, and the segmentation results are ranked based on word frequency. The final power-related dictionary was created by having professionals with expertise in the power field revise technical terms, colloquialisms, word strings, and phrases.

4. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 1, characterized in that, The text encoder is based on the BERT architecture and is pre-trained for domain adaptation on power industry literature, equipment manuals, and fault report data.

5. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 1, characterized in that, The visual encoder uses a lightweight visual Transformer to extract image features, including device appearance, location, and connectivity.

6. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 1, characterized in that, The specific implementation methods of the fusion layer include: S1, Alignment Mechanism: Using a learnable mapping matrix and Text features and visual features Mapping to a unified semantic space: S2, Cross Attention: Calculates the interaction between text and visual features to generate fused features; Calculate the attention weights of text to vision. visual attention weights to text : ， in Q, K The query vector and key vector are generated through a linear transformation; The characteristics after fusion are: ， S3, Weighted Summation: Dynamically adjust weights based on the importance of modal features. α，β To form the final fusion characteristics ; 。 7. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 1, characterized in that, The cascaded binary annotation framework CasRel includes: S1. Entity Recognition: Fused Features Apply a cascaded binary annotation framework; Head entity recognition: Using a linear layer and sigmoid activation function, determine whether each token is the start / end position of the head entity; Tail entity recognition: Based on the head entities, identify the tail entities; generate a candidate head entity set. ; S2, Joint Identification of Relationships and Tail Entities: For each The dual affine attention mechanism is used to calculate its relationship with all candidate tail entities e. t Relationship probability: ， in W r For relational type parameter matrix, σ The sigmoid function is used to select the relation type with the highest probability and its corresponding tail entity to form a triple. ; S3. Knowledge Graph Construction: Constructing Triples Stored in the Neo4j graph database to form a knowledge graph.

8. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 7, characterized in that, The dual affine attention mechanism is as follows: Head entity Tail-end entity Features Perform affine transformation: ， Constructing the feature interaction matrix : ， Extracting relational features using max pooling: 。 9. The method for extracting joint relationships of power entities based on a multimodal large model according to claim 1, characterized in that, The conflict detection module uses SHACL and TBox content to perform consistency checks on newly extracted relationships and identify whether there are conflicting relationships. When a conflict is detected between a newly extracted relationship and existing knowledge, the conflict resolution mechanism is triggered.

10. A power entity joint relationship extraction system based on a multimodal large model, characterized in that, It adopts the power entity joint relationship extraction method based on a multimodal large model as described in any one of claims 1-9. By inputting the collected power text and power equipment images into the trained power entity joint relationship extraction model, entity-relationship triples and a visualized knowledge graph are obtained.

Images