A multi-modal knowledge graph construction method for relay protection

Abstract

This invention relates to the field of power system technology, specifically to a method for constructing a multimodal knowledge graph for relay protection. The method includes the following steps: S1, constructing a multimodal data sample library, including acquiring multimodal data related to relay protection, preprocessing the multimodal data, and then labeling it to establish a standardized labeling system; S2, multimodal knowledge extraction, including knowledge extraction for textual data and knowledge extraction for graphical data; S3, alignment and fusion of multimodal knowledge; S4, constructing a multimodal knowledge graph for relay protection based on a triplet topology link structure. This method integrates multimodal knowledge such as text and images, achieving structured and systematic knowledge management, and providing reliable knowledge support for model construction and extended applications and risk identification of relay protection setting verification and management operations.

Description

Technical Field

[0001] This invention relates to the field of power system technology, specifically to a method for constructing a multimodal knowledge graph for relay protection. Background Technology

[0002] Relay protection is the "first line of defense" in a power system, undertaking the critical tasks of fault detection, isolation, and isolation. Its operational reliability directly determines the safety level of the power system. Setting, verification, and management are core components ensuring the safe and stable operation of power equipment. The quality of its work directly affects the rapid isolation of grid faults, the effective avoidance of equipment damage, and the continuous reliability of power supply.

[0003] Against the backdrop of the accelerated construction of new power systems, the continuous expansion of power grid scale, and the increasingly complex operating characteristics, the shortcomings and bottlenecks currently faced by this business are becoming increasingly prominent. Among them, the problems exposed in equipment parameter acquisition and power grid topology modeling, as the core foundation of setting calculation, are particularly prominent and have become key factors restricting the efficient development of the business and affecting the accuracy and timeliness of setting calculation.

[0004] Currently, knowledge management for relay protection equipment parameter acquisition and power grid topology modeling still relies mainly on traditional document archiving and manual organization, which presents several prominent challenges: First, multimodal knowledge is stored in a scattered manner, with text and graphical knowledge being fragmented, making it difficult to achieve cross-modal knowledge association and retrieval, resulting in low knowledge reuse rates; second, the entities and relationships in core documents such as relay protection model construction and its extended applications are complexly nested, making it difficult to achieve accurate and structured information extraction using only a single entity type recognition technology, thus failing to provide effective support for intelligent applications; third, knowledge from different sources suffers from semantic ambiguity and inconsistent expression, making knowledge integration difficult and hindering the formation of a structured and systematic knowledge system, which restricts the intelligent development of relay protection setting, verification, and management operations. Summary of the Invention

[0005] To address the technical challenges of fragmented knowledge, low correlation, and complex nesting of entities and relationships in relay protection systems' setting verification and management scenarios, which involve multiple modalities such as text and images, making it difficult to meet the practical needs of intelligent model construction and risk identification, this invention provides a multimodal knowledge graph construction method for relay protection. This method integrates multimodal knowledge such as text and images, enabling structured and systematic knowledge management, and providing reliable knowledge support for model construction, extended applications, and risk identification in relay protection setting verification and management scenarios.

[0006] The technical solution adopted in this invention is to provide a method for constructing a multimodal knowledge graph for relay protection, including the following steps: S1. Construction of a multimodal data sample library, including acquiring multimodal data related to relay protection, and labeling the multimodal data after preprocessing to establish a standardized labeling system. The multimodal data includes text data and graphic data. The text data includes operation procedures, equipment parameters, and risk point annotations. The graphic data includes information flow diagrams, relay protection setting configuration diagrams, and primary system diagrams. S2. Multimodal knowledge extraction includes entity and event recognition methods based on a general information extraction framework for text data, introducing structured extraction language and structural pattern indicators, and combining an encoder with knowledge fusion and enhanced representation and a dual-pointer decoder for knowledge extraction; for graphic data, OCR technology is used to recognize text information in graphic data and convert it into editable structured text, and the YOLO algorithm is used to identify key device entities and connection primitives in graphic data. Each primitive is treated as a node of the graph, and the relationships and positional relationships between primitives are treated as edges of the graph, constructing a graph data structure. The graph data structure is input into the GCN model, and the relationships between primitives are mined through graph convolution operations. S3. Alignment and fusion of multimodal knowledge; S4. Based on the triplet topology link structure, construct a multimodal knowledge graph for relay protection.

[0007] The data preprocessing described in step S1 includes data cleaning, text typo correction, data normalization, word segmentation, and stop word removal for text data; and format normalization, image enhancement, and data augmentation for graphic data.

[0008] The tagging system described in step S1 includes entity tags, relationship tags, and event tags for text data; the entity tags include device entities, operation entities, and terminology entities; the relationship tags include the association between devices, the association between devices and operations, and the association between operations and risks; and the event tags include device construction events, risk identification events, and safety measure implementation events. For graphic data, there are element labels, relationship labels, and location labels; the element labels include device elements, text elements, and connection elements, the relationship labels include the association relationships between elements, and the location labels include the positional relationships between elements.

[0009] The encoder for knowledge fusion enhancement described in step S2 uses BERT as its base model, incorporating knowledge from the relay protection domain. This domain knowledge is transformed into vector form and fused with text feature vectors to form enhanced semantic feature vectors. The fusion formula is as follows. Formula 1 In Equation 1, This represents the enhanced semantic feature vector of the i-th token after fusion; This represents the text feature vector of the i-th token output by the BERT model; This represents the domain knowledge vector corresponding to the i-th token; This represents the attention weight, used to adjust the fusion ratio of text features and domain knowledge features. The calculation formula is as follows: Formula 2 In Equation 2, n represents the number of tokens in the text.

[0010] The dual-pointer decoder described in step S2 introduces a CRF model for constraint at the output layer, and uses a constructed text-based data sample library to train and optimize the dual-pointer decoder.

[0011] During training, the dual-pointer decoder uses the cross-entropy loss function as the model's loss function and employs the Adam optimizer to optimize the dual-pointer decoder parameters. This optimizer measures the difference between the model's prediction results and the actual labeled results. The loss function formula is as follows: Formula 3 In Equation 3, N Indicates the number of training samples; M Indicates the number of tags; Indicates the first i The first sample j The true value of each label, either 0 or 1; The model predicts the first... i The first sample j The probability that a label is 1.

[0012] The GCN model described in step S2 captures the relationships between nodes by aggregating the features of their neighbors and updating the feature representations of those nodes. The formula is as follows. Formula 4 In Equation 4, Indicates the first l Nodes in a layered network i eigenvectors, Represents a node i The set of neighboring nodes; Represents a node i The number of neighboring nodes; Indicates the first l The weight matrix of the layer network; Indicates the first l The bias vector of the layer network; This indicates the activation function that uses the ReLU function; An attention mechanism is introduced to weight the contributions of different neighboring nodes. The formula for calculating the attention weight is as follows: Formula 5 In Equation 5, Represents a node j For nodes i Attention weights; Represents the attention weight matrix; The formula for the improved GCN model that incorporates an attention mechanism is as follows: Formula 6.

[0013] The multimodal knowledge alignment and fusion described in step S3 includes the alignment of the names of devices in the primary system diagram and the names in the relay protection setting configuration diagram. The Sentence-BERT model is used to perform knowledge vectorization processing on the device setting names in the relay protection setting configuration diagram. The devices are grouped according to the differences in the type of device names, and semantic encoding is performed on different types of device names. The device names are converted into 128-dimensional semantic vectors, and multiple vector databases corresponding to different device types are constructed. The FastText algorithm is used to classify the primitives in a primary system diagram. The device primitives detected in the primary system diagram are used as classification objects and classified according to device type. Based on the classification results of the system diagram elements, the equipment type corresponding to each equipment element is determined. Semantic retrieval is performed in the equipment type vector database of the corresponding relay protection setting configuration diagram. The similarity between the text information corresponding to the equipment element and the name vector of the relay protection setting configuration diagram is calculated. The relay protection setting configuration diagram name with the highest similarity and exceeding the threshold of 0.8 is associated and aligned with the equipment element.

[0014] The multimodal knowledge alignment and fusion described in step S3 includes knowledge fusion. The specific process involves first performing semantic normalization of knowledge, then assigning different weights to knowledge from different modalities, and finally performing weighted fusion to form a structured and systematic multimodal knowledge set. The formula for calculating the weights is as follows: Formula 7 In Equation 7, Indicates the first k Weights of modal knowledge; Indicates the first k The accuracy of extracting modal knowledge; Indicates the first k The correlation of modal knowledge, with a value range of 0 to 1; m This represents the number of types of modal knowledge.

[0015] The specific process of constructing a multimodal knowledge graph for relay protection as described in step S4 includes: vectorizing the knowledge units constructed from the fused relay protection model; converting the knowledge units into 128-dimensional semantic vectors using the Sentence-BERT model; transforming discrete knowledge units into high-dimensional semantic vectors; and achieving multidimensional data similarity matching by calculating the distance between different knowledge units in the semantic space. The semantic distance is calculated using Euclidean distance, as shown in the following formula. Formula 8 In Equation 8, u , v A semantic vector representing two knowledge units; , Represents the first of two vectors i One component; n Indicates the dimension of the vector; By calculating semantic distance, knowledge units with a semantic distance less than a threshold of 0.5 are associated to obtain semantic information. Using the "entity-relationship-entity" triple as the core, the extracted text-based knowledge and image-based knowledge are topologically linked to construct a dynamic semantic association network; After the comprehensive multimodal knowledge graph is generated, all knowledge elements in the dynamic semantic association network are integrated to generate a comprehensive multimodal knowledge graph for relay protection. The knowledge graph consists of an entity layer, a relationship layer, and an event layer: the entity layer includes equipment entities, operation entities, risk entities, and terminology entities; the relationship layer includes the relationships between equipment, between equipment and operation, between operation and risk, and between graph elements; the event layer includes equipment operation events, risk identification events, and safety measure implementation events.

[0016] The beneficial effects of this invention are as follows: It provides a method for constructing a multimodal knowledge graph for relay protection. This method collects historical records, model construction, and extended applications related to relay protection, as well as textual data, information flow graphs, relay protection setting configuration diagrams, and primary system diagrams. A multimodal data sample library is constructed, and the initial sample library is standardized through normalization, data augmentation, and labeling. Addressing the complex nesting of entities and relationships in relay protection model construction and its extended applications, the invention employs an entity and event recognition method based on a general information extraction framework (UIE). This method unifies the model input and output structure through structured language extraction and structural pattern indicators, and combines an encoder with knowledge fusion-enhanced representation and a dual-pointer decoder to achieve accurate entity and event extraction, providing core knowledge elements for knowledge graph construction. For image data, a comprehensive processing scheme based on OCR, YOLO, and GCN is proposed. A primary system diagram data labeling system is established, using OCR technology to recognize image text content, combining the YOLO algorithm to detect key entities and relationships in the image, and using deep learning encoding to realize the identification of entities, relationships, and events. Intelligent extraction: Combining GCN technology to annotate primary wiring diagrams, achieving accurate extraction of element relationships and positional relationships, enabling the collection and structured parsing of equipment operating status and test data, and simultaneously completing data cleaning and verification. Based on the input data and rule engine, a high-precision electrical system simulation model is automatically generated. The Sentence-BERT model is used to group equipment names in the relay protection setting configuration diagram by type and vectorize them into knowledge, constructing multiple vector databases. FastText is used to classify the elements of the primary system diagram, and retrieval is performed in the corresponding vector database based on the element category, achieving the association and alignment between the equipment in the primary system diagram and the names in the relay protection setting configuration diagram. Finally, the knowledge units of the relay protection model are vectorized, and the similarity matching of multi-dimensional data is achieved by calculating the semantic space distance. Knowledge alignment and weighted fusion technology are used to construct a model and its extended applications to unify the representation of equipment, terminology, and rule information. Based on triplet topology linking technology, a dynamic semantic association network containing historical data, equipment parameters, and equipment image sources is constructed, ultimately generating a comprehensive multimodal knowledge graph to support intelligent compilation and risk identification. The design of this method can effectively reduce errors and time costs associated with manual modeling, providing efficient and accurate data support and model foundation for relay protection setting and verification. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] like Figure 1 As shown, this invention provides a method for constructing a multimodal knowledge graph for relay protection, including the following steps: S1. Construction of a multimodal data sample library The relay protection model is constructed using relevant historical data as the core data source, and multiple types and modal data are collected. After preprocessing and labeling, a standardized multimodal data sample library is constructed.

[0019] Based on the practical needs of relay protection model construction and identification, the data mentioned in this method covers two major categories: text and graphics. It comprehensively covers the core knowledge of the entire relay protection model construction process. Specific data types mainly include historical records, model construction and its extended application data, covering the operation process, equipment parameters and risk point annotations of relay protection model construction; information flow diagrams, relay protection setting configuration diagrams, and primary system diagrams. The information flow diagrams reflect the signal transmission paths and logical relationships in the relay protection system; the relay protection setting configuration diagrams cover the equipment settings and connection methods of relay protection; and the primary system diagrams cover the main wiring methods, equipment distribution and correlation relationships of the power system.

[0020] Because the collected multimodal data suffers from inconsistent formats, data redundancy, and noise interference, it cannot be directly used for knowledge extraction and graph construction. Therefore, it is necessary to perform preprocessing operations on various types of data, mainly including two core steps: data normalization and data augmentation, to eliminate data heterogeneity.

[0021] For text-based data, preprocessing includes data cleaning, removing invalid records, duplicate data, and redundant information; correcting typos and formatting errors to ensure accuracy; and ensuring text normalization by converting text data in different formats such as Word, PDF, and TXT to TXT format and encoding it uniformly to UTF-8. Simultaneously, word segmentation and stop word removal are performed, using a word segmentation tool to segment the text while preserving core semantic information. Furthermore, for categories with limited text data, data augmentation is achieved through synonym replacement, sentence rearrangement, and text summarization to enhance the diversity of the text data and meet the needs of subsequent model training.

[0022] For graphical data, preprocessing includes format normalization, converting graphical data of different formats to PNG format to eliminate the impact of format differences, and image enhancement to improve image clarity, addressing issues such as blurriness, noise, and uneven lighting. The graphical data is augmented using rotation, flipping, translation, and scaling to increase the sample size and prevent overfitting during subsequent model training. For example, a system graph is rotated by 0°, 90°, 180°, and 270°, flipped horizontally and vertically, and scaled by 0.5 to 2 times to generate more training samples and improve the model's generalization ability.

[0023] To achieve accurate extraction of multimodal knowledge, it is necessary to label the preprocessed multimodal data, establish a standardized labeling system, clarify the classification standards and labeling rules of various knowledge elements, and cover the core knowledge elements of text and graphic data.

[0024] For text-based data, the tagging system mainly includes three categories: entity tags, relationship tags, and event tags. Entity tags mainly include equipment entities, such as transformers, switches, and busbars; operation entities, such as closing, opening, and grounding; risk entities, such as load side, electrical node connections, and physical topology node connections; and terminology entities, such as model construction and its extended applications. Protection setting relationship tags mainly include the relationships between equipment, such as series, parallel, and control; the relationships between equipment and operation, such as operation objects and operation methods; and the relationships between operation and risk, such as operation leading to risk and risk corresponding to operation. Event tags mainly include equipment construction events, risk identification events, and safety measure implementation events.

[0025] For graphical data, the labeling system mainly includes three categories: element labels, relationship labels, and location labels. Element labels mainly include equipment elements, such as switches, busbars, transformers, and lines; text elements, such as equipment names and parameter annotations; and connection elements, such as wires and cables. Relationship labels include the relationships between elements, such as side switches and busbars, or side switches and center switches. Location labels include the positional relationships between elements, such as up / down, left / right, containment, and adjacent. Among these, the labeling system for primary system diagrams covers equipment elements, connection elements, and various relationships and locational relationships; the labeling system for relay protection setting configuration diagrams covers equipment settings and wiring relationships; and the labeling system for information flow diagrams covers signal transmission paths and logical relationships, clearly defining the location, type, and relationship of elements to ensure the accuracy of labeling.

[0026] Through the above data collection, preprocessing, and labeling operations, a multimodal data sample library for risk identification in relay protection model construction was built, providing high-quality and standardized data support for subsequent knowledge extraction, fusion, and knowledge graph construction.

[0027] S2, Multimodal Knowledge Extraction The purpose of multimodal knowledge extraction is to extract knowledge elements such as entities, relationships, and events from multimodal data such as text and images, so as to provide a foundation for the construction of knowledge graphs.

[0028] To address the different characteristics of textual and graphical data, corresponding knowledge extraction methods were designed to achieve comprehensive and accurate extraction of multimodal knowledge. Textual data, such as model construction and its extended applications, and historical records, are important carriers of knowledge for relay protection model construction, containing a large amount of entity, relation, and event information. However, the core documents of relay protection model construction and its extended applications contain complex nested entities and relations, making it difficult to achieve accurate and structured information extraction using only a single entity type recognition technique. To address this issue, this paper proposes an entity and event recognition method based on a Universal Information Extraction Framework (UIE) to achieve accurate extraction of textual knowledge.

[0029] S21. Knowledge Extraction from Text-Based Data The Universal Information Extraction Framework (UIE) features a unified input-output structure, enabling unified modeling for various information extraction tasks such as entity recognition, relation extraction, and event extraction. However, its accuracy in handling nested entities and complex relationships still needs improvement. This method, tailored to the characteristics of relay protection text data, improves the UIE framework by introducing a structured extraction language and a structure pattern indicator. It combines a knowledge fusion-enhanced representation encoder and a dual-pointer decoder to enhance the accuracy of entity and event recognition.

[0030] Structured extraction language is used to standardize relay protection text data, converting unstructured text data into a structured input format, clarifying the boundaries and semantic relationships of entities, relationships, and events in the text, and unifying the input structure of the model. Structure pattern indicators are used to identify key structural information in the text, such as equipment names, operating instructions, and risk descriptions, helping the model quickly locate core knowledge elements and reduce interference from irrelevant information.

[0031] The knowledge fusion-enhanced representation encoder is the core of the improved UIE framework. Its purpose is to strengthen the semantic representation capability of text features and improve the recognition accuracy of nested entities and complex relationships. This encoder uses BERT as the base model and incorporates knowledge from the relay protection domain for fusion. The domain knowledge is transformed into vector form and fused with the text feature vector to form an enhanced semantic feature vector. Specifically, first, a relay protection domain knowledge dictionary is constructed, containing core domain knowledge such as equipment names, technical terms, and operating rules. Then, the Word2Vec model is used to vectorize the domain knowledge dictionary to obtain domain knowledge vectors. Finally, an attention mechanism is used to fuse the domain knowledge vectors with the text feature vectors output by the BERT model. The fusion formula is as follows: Formula 1 In Equation 1, This represents the enhanced semantic feature vector of the i-th token after fusion; This represents the text feature vector of the i-th token output by the BERT model; This represents the domain knowledge vector corresponding to the i-th token; This represents the attention weight, used to adjust the fusion ratio of text features and domain knowledge features.

[0032] The formula for calculating attention weights is as follows: Formula 2 In Equation 2, n represents the number of tokens in the text; By using the above fusion method, the semantic representation of text features can be effectively enhanced, enabling the model to better understand the professional semantics in the field of relay protection and improve the recognition accuracy of nested entities and complex relationships.

[0033] A dual-pointer decoder is used to achieve accurate entity and event extraction. It uses two pointers to locate the start and end positions of an entity or event, effectively solving the problem of identifying nested and overlapping entities. This decoder introduces a Conditional Random Field (CRF) model into the output layer to optimize the pointer positioning results and improve the accuracy of the recognition. Specifically, the decoder outputs the start and end position probabilities of an entity or event. The CRF model constrains the position sequence, ensuring that the start position is less than the end position and that the entity or event type is accurately labeled, ultimately outputting accurate entity and event recognition results. To ensure the recognition performance of the improved UIE model, a constructed text-based data sample library is used for training and optimization. During training, the cross-entropy loss function is used as the model's loss function to measure the difference between the model's prediction results and the actual labeled results. The loss function formula is as follows: Formula 3 In Equation 3, N Indicates the number of training samples; M Indicates the number of tags; Indicates the first i The first sample j The true value of each label is 0 or 1; The model predicts the first... i The first sample j The probability that a label is 1.

[0034] During training, the Adam optimizer was used to optimize the model parameters, with a learning rate of 1e-5, a batch size of 16, and 50 training epochs. The optimal model parameters were then saved. Simultaneously, data augmentation and regularization were employed to further optimize the model and improve its generalization ability.

[0035] S22 Knowledge Extraction from Graphical Data Graphical data, such as primary system diagrams, relay protection setting configuration diagrams, and information flow diagrams, are important components of the knowledge required for relay protection model construction. These data contain a large amount of information about equipment elements, connections, and locations, which is difficult to extract in a structured manner manually. To address this issue, this paper proposes a comprehensive technical solution based on OCR, YOLO, and GCN to conduct research on the intelligent recognition and extraction of graphical knowledge.

[0036] First, for text content such as equipment names and parameter annotations in graphic data, OCR technology is used for accurate recognition, converting the text information in the image into editable structured text, solving the problem of the difficulty in directly extracting text from images. This method uses a CNN+LSTM-based OCR model, which can effectively extract text features from images and achieve accurate text recognition. Specifically, firstly, the CNN model extracts features from the image to obtain feature maps of the text regions; then, the LSTM model performs sequence modeling on the feature maps to capture the contextual relationships of the text; finally, the CTC connectionist temporal classification decoder decodes the sequence features to obtain the recognized text content. To improve the accuracy of OCR recognition, the graphic data is preprocessed, such as image enhancement and text region localization. Simultaneously, transfer learning is used to fine-tune the pre-trained OCR model on the constructed graphic text dataset, improving the model's ability to recognize relay protection graphic text.

[0037] Secondly, for key equipment entities and connection elements in graphic data, the YOLO algorithm is used for efficient detection, quickly locating entity positions and identifying entity types. The YOLO algorithm has the advantages of real-time detection and high accuracy, and can effectively adapt to the characteristics of large number and dense distribution of entities in graphic data. This method uses the YOLOv8 model, which has strong advantages in both detection speed and accuracy. During training, a constructed graphic data sample library is used as the training set to train the YOLOv8 model, optimize model parameters, and improve the model's detection accuracy of elements in relay protection graphics. Specifically, the elements in the graphic data are divided into three categories: equipment elements, connection elements, and text elements. Each category of elements is labeled, and then the labeled data is input into the YOLOv8 model for training. The learning rate is set to 1e-4, the batch size is set to 8, and the number of training rounds is set to 30. After training, the model is used to detect graphic data, outputting the position coordinates, type, and confidence score of the elements. The confidence score threshold is set to 0.5 to ensure the accuracy of the detection results.

[0038] Building upon image-text recognition and primitive detection, further exploration of the relationships and positions between primitives is needed to provide graph-based relational knowledge for knowledge graph construction. This method combines a GCN graph convolutional network for deep mining of primitive relationships. The detected primitives are modeled to construct graph structure data. Each primitive is treated as a node in the graph, and the relationships and positional relationships between primitives are treated as edges. ,in V Represents a set of primitive nodes. E This represents the set of edges. For each node... Its eigenvectors It consists of primitive type features, positional features, and text features. The text features are the text obtained by OCR recognition, and the feature vector has a dimension of 128. For each edge... The weights are determined by the distance and the degree of association between graph primitives. The constructed graph structure data is input into the GCN model, and potential relationships between graph primitives are mined through graph convolution operations. The GCN model captures the relationships between nodes by aggregating the features of their neighbors and updating the feature representations of the nodes. The operation is as follows: Formula 4 In Equation 4, Indicates the first l Nodes in a layered network i eigenvectors, Represents a node i The set of neighboring nodes; Represents a node i The number of neighboring nodes; Indicates the first l The weight matrix of the layer network; Indicates the first l The bias vector of the layer network; This indicates the activation function using the ReLU function. This method improves the GCN model by incorporating the characteristics of relay protection graphs and introducing an attention mechanism to weight the contributions of different neighboring nodes, thereby enhancing the accuracy of relation extraction.

[0039] The formula for calculating attention weights is as follows: Formula 5 In Equation 5, Represents a node j For nodes i Attention weights; This represents the attention weight matrix. The improved operation is as follows: Formula 6 The improved GCN model described above can accurately capture the relationships and positions between graphic elements, achieving precise extraction of element relationships. This method, combined with a constructed graphic data sample library, completes the annotation of a primary wiring diagram and uses the improved GCN model to accurately extract six types of element relationships and four types of element positional relationships. The six element relationships are: the relationship between the side switch and the busbar; the relationship between the side switch and the middle switch; the relationship between the middle switch and the transformer; the relationship between the side switch and the transformer switch; the relationship between the line and the side switch; and the relationship between the line and the middle switch. The four element positional relationships are: vertical relationship, horizontal relationship, containment relationship, and adjacent relationship.

[0040] S3. Alignment and Fusion of Multimodal Knowledge Since textual and graphical knowledge come from different data sources, there are problems such as semantic ambiguity, inconsistent expression, and heterogeneous knowledge. Knowledge alignment and fusion are needed to achieve a unified representation of multimodal knowledge and lay the foundation for the construction of knowledge graphs. This method designs a step-by-step knowledge alignment and fusion scheme, which is divided into two parts: cross-modal knowledge alignment and knowledge fusion.

[0041] S31, Cross-modal knowledge alignment The core of cross-modal knowledge alignment is to achieve semantic association between textual and graphical knowledge. This method focuses on aligning the names of devices in the primary system diagram with those in the relay protection setting configuration diagram. First, the device setting names in the relay protection setting configuration diagram are processed into knowledge vectors. The Sentence-BERT model is adopted, which can convert text into vectors with semantic representation capabilities and effectively capture the semantic features of the text, improving the accuracy of the vector representation. Device names are grouped according to their type, such as line protection, transformer protection, and circuit breaker protection. Semantic encoding is performed on different types of device names, converting them into 128-dimensional semantic vectors, and then constructing multiple vector databases corresponding to different device types. For example, device names of protection types, such as main transformer protection and line protection, are encoded to construct a main transformer protection vector database; line protection names are encoded to construct a line protection vector database, ensuring the vector distinguishability of different types of device names. Secondly, the primitives in the primary system diagram are classified using the FastText algorithm, which boasts advantages such as fast training speed and high classification accuracy, effectively achieving rapid primitive classification. Equipment primitives detected in the primary system diagram are used as classification objects, categorized according to equipment type, such as switches, busbars, transformers, and lines. The FastText classification model is then trained to achieve automatic primitive type classification. During training, primitive annotation data from a constructed graphical data sample library is used as the training set, with the primitive feature vectors as input and the primitive type as output. Finally, the association and alignment of the names of equipment in the primary system diagram with those in the relay protection setting configuration diagram are implemented.

[0042] Based on the classification results of the system diagram elements, the equipment type corresponding to each equipment element is determined. Then, semantic retrieval is performed in the equipment type vector database of the corresponding relay protection setting configuration diagram. The similarity between the text information corresponding to the equipment element and the name vector of the relay protection setting configuration diagram is calculated. The text information is the equipment name recognized by OCR. The relay protection setting configuration diagram name with the highest similarity and exceeding the threshold of 0.8 is associated and aligned with the equipment element.

[0043] Semantic similarity is calculated using cosine similarity: Formula 9 In Equation 9, u A semantic vector representing the device name; v A semantic vector representing the name of the relay protection setting configuration diagram; u * v Represents the dot product of two vectors; | u |、| v | represents the magnitudes of the two vectors respectively.

[0044] Through the above-mentioned association and alignment process, the names of the primary system diagram devices and the relay protection setting configuration diagram are accurately associated, which breaks down the semantic connection between text-based knowledge and graphical knowledge and lays the foundation for multimodal knowledge fusion.

[0045] S32, Knowledge Integration The core of knowledge fusion is to eliminate redundant information and semantic conflicts in multimodal knowledge, achieve a unified representation of equipment, terminology, and rule information, and improve the consistency and reusability of knowledge. This method employs knowledge alignment and weighted fusion techniques for multimodal knowledge fusion. First, knowledge semantic standardization is performed. Addressing issues such as ambiguity in equipment names, inconsistent terminology, and differences in rule expressions within multimodal knowledge, a knowledge dictionary for the relay protection field is established to clarify standardized expressions for equipment names, professional terms, and operating rules, and to standardize and align the extracted multimodal knowledge. For example, the equipment name "side switch" is standardized as "side circuit breaker"; the term "model construction" is standardized as "relay protection model construction and its extended applications," eliminating semantic ambiguity and ensuring knowledge consistency. Then, weighted knowledge fusion is performed. Since knowledge from different sources and in different modalities has varying reliability and importance, a weighted fusion strategy is needed to integrate multimodal knowledge.

[0046] This method assigns different weights to different modalities of knowledge based on factors such as the source, reliability, and accuracy of the knowledge. The formula for calculating the weights is as follows: Formula 7 In Equation 7, Indicates the first k Weights of modal knowledge; Indicates the first k The accuracy of extracting modal knowledge; Indicates the first k The correlation between modal knowledge and the degree of correlation between relay protection model construction and risk identification, with a value range of 0 to 1; m This represents the number of types of modal knowledge.

[0047] Finally, the merged knowledge is verified and corrected. The merged knowledge is reviewed manually to correct errors and supplement missing knowledge elements, ensuring its accuracy and completeness. Through knowledge fusion, a structured and systematic multimodal knowledge set is formed, providing unified knowledge support for the construction of knowledge graphs.

[0048] S4. Based on the triplet topology link structure, construct a multimodal knowledge graph for relay protection.

[0049] Building upon multimodal knowledge extraction, alignment, and fusion, this method constructs a multimodal knowledge graph for relay protection based on triplet topology linking technology, enabling structured knowledge management and intelligent application. The constructed multimodal knowledge graph uses "entity-relationship-entity" triples as its core, encompassing both textual and graphical multimodal knowledge. The fused relay protection model's knowledge units are vectorized, including entities, relations, and events. The Sentence-BERT model is used to convert these knowledge units into 128-dimensional semantic vectors, transforming discrete knowledge units into high-dimensional semantic vectors, laying the foundation for subsequent semantic similarity matching and topology linking. By calculating the distance between different knowledge units in the semantic space, multidimensional data similarity matching is achieved, uncovering potential semantic associations between multi-source knowledge. Semantic distance is calculated using Euclidean distance. Formula 8 In Equation 8, u , v A semantic vector representing two knowledge units; , Represents the first of two vectors i One component; n Indicates the dimension of the vector.

[0050] By calculating semantic distance, knowledge units with a semantic distance less than a threshold of 0.5 are associated to uncover potential semantic relationships.

[0051] Employing triplet topology linking technology, with "entity-relationship-entity" triples as the core, this method topologically associates pre-processed historical records, model constructions, and their extended applications with graphical knowledge extracted from primary system diagrams, relay protection setting configuration diagrams, and information flow diagrams to construct a dynamic semantic association network. This network can update knowledge elements and relationships in real time. When new model data emerges, such as newly added model constructions and their extended applications, or new primary system diagrams, it can quickly extract knowledge elements and update the nodes and edges in the network, achieving dynamic updates to the knowledge graph.

[0052] After the comprehensive multimodal knowledge graph is generated, all knowledge elements in the dynamic semantic association network are integrated to generate a comprehensive multimodal knowledge graph for relay protection. This knowledge graph consists of three core layers: entity layer, relationship layer, and event layer. The entity layer covers various entities such as equipment entities, operation entities, risk entities, and terminology entities; the relationship layer covers various relationships such as between equipment, between equipment and operation, between operation and risk, and between graph elements; the event layer covers various events such as equipment operation events, risk identification events, and safety measure implementation events.

Claims

1. A method for constructing a multimodal knowledge graph for relay protection, characterized in that: Includes the following steps, S1. Construction of a multimodal data sample library, including acquiring multimodal data related to relay protection, and labeling the multimodal data after preprocessing to establish a standardized labeling system. The multimodal data includes text data and graphic data. The text data includes operation procedures, equipment parameters, and risk point annotations. The graphic data includes information flow diagrams, relay protection setting configuration diagrams, and primary system diagrams. S2, Multimodal knowledge extraction, including entity and event recognition methods based on a general information extraction framework for text data, introducing structured extraction language and structural pattern indicators, and combining an encoder with knowledge fusion and enhanced representation and a dual-pointer decoder for knowledge extraction; For graphical data, OCR technology is used to recognize text information in the graphical data and convert it into editable structured text. The YOLO algorithm is used to identify key device entities and connecting primitives in the graphical data. Each primitive is treated as a node of the graph, and the relationships and positional relationships between primitives are treated as edges of the graph. The graph data structure is then input into the GCN model, and the relationships between primitives are mined through graph convolution operations. S3. Alignment and fusion of multimodal knowledge; S4. Based on the triplet topology link structure, construct a multimodal knowledge graph for relay protection.

2. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The data preprocessing described in step S1 includes data cleaning, text typo correction, data normalization, word segmentation, and stop word removal for text data; and format normalization, image enhancement, and data augmentation for graphic data.

3. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The tagging system described in step S1 includes entity tags, relationship tags, and event tags for text data; the entity tags include device entities, operation entities, and terminology entities; the relationship tags include the association between devices, the association between devices and operations, and the association between operations and risks; and the event tags include device construction events, risk identification events, and safety measure implementation events. For graphic data, there are element labels, relationship labels, and location labels; the element labels include device elements, text elements, and connection elements, the relationship labels include the association relationships between elements, and the location labels include the positional relationships between elements.

4. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The encoder for knowledge fusion enhancement described in step S2 uses BERT as its base model, incorporating knowledge from the relay protection domain. This domain knowledge is transformed into vector form and fused with text feature vectors to form enhanced semantic feature vectors. The fusion formula is as follows. Formula 1 In Equation 1, This represents the enhanced semantic feature vector of the i-th token after fusion; This represents the text feature vector of the i-th token output by the BERT model; This represents the domain knowledge vector corresponding to the i-th token; This represents the attention weight, used to adjust the fusion ratio of text features and domain knowledge features. The calculation formula is as follows: Formula 2 In Equation 2, n represents the number of tokens in the text.

5. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The dual-pointer decoder described in step S2 introduces a CRF model for constraint at the output layer, and uses a constructed text-based data sample library to train and optimize the dual-pointer decoder.

6. The method for constructing a multimodal knowledge graph for relay protection according to claim 5, characterized in that: During training, the dual-pointer decoder uses the cross-entropy loss function as the model's loss function and employs the Adam optimizer to optimize the dual-pointer decoder parameters. This optimizer measures the difference between the model's prediction results and the actual labeled results. The loss function formula is as follows: Formula 3 In Equation 3, N Indicates the number of training samples; M Indicates the number of tags; Indicates the first i The first sample j The true value of each label, either 0 or 1; The model predicts the first... i The first sample j The probability that a label is 1.

7. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The GCN model described in step S2 captures the relationships between nodes by aggregating the features of their neighbors and updating the feature representations of those nodes. The formula is as follows. Formula 4 In Equation 4, Indicates the first l Nodes in a layered network i eigenvectors, Represents a node i The set of neighboring nodes; Represents a node i The number of neighboring nodes; Indicates the first l The weight matrix of the layer network; Indicates the first l The bias vector of the layer network; This indicates the activation function that uses the ReLU function; An attention mechanism is introduced to weight the contributions of different neighboring nodes. The formula for calculating the attention weight is as follows: Formula 5 In Equation 5, Represents a node j For nodes i Attention weights; Represents the attention weight matrix; The formula for the improved GCN model that incorporates an attention mechanism is as follows: Formula 6.

8. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The multimodal knowledge alignment and fusion described in step S3 includes the alignment of the names of devices in the primary system diagram and the names in the relay protection setting configuration diagram. The Sentence-BERT model is used to perform knowledge vectorization processing on the device setting names in the relay protection setting configuration diagram. The devices are grouped according to the differences in the type of device names, and semantic encoding is performed on different types of device names. The device names are converted into 128-dimensional semantic vectors, and multiple vector databases corresponding to different device types are constructed. The FastText algorithm is used to classify the primitives in a primary system diagram. The device primitives detected in the primary system diagram are used as classification objects and classified according to device type. Based on the classification results of the system diagram elements, the equipment type corresponding to each equipment element is determined. Semantic retrieval is performed in the equipment type vector database of the corresponding relay protection setting configuration diagram. The similarity between the text information corresponding to the equipment element and the name vector of the relay protection setting configuration diagram is calculated. The relay protection setting configuration diagram name with the highest similarity and exceeding the threshold of 0.8 is associated and aligned with the equipment element.

9. The method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The multimodal knowledge alignment and fusion described in step S3 includes knowledge fusion. The specific process involves first performing semantic normalization of knowledge, then assigning different weights to knowledge from different modalities, and finally performing weighted fusion to form a structured and systematic multimodal knowledge set. The formula for calculating the weights is as follows: Formula 7 In Equation 7, Indicates the first k Weights of modal knowledge; Indicates the first k The accuracy of extracting modal knowledge; Indicates the first k The correlation of modal knowledge, with a value range of 0 to 1; m This represents the number of types of modal knowledge.

10. A method for constructing a multimodal knowledge graph for relay protection according to claim 1, characterized in that: The specific process of constructing a multimodal knowledge graph for relay protection as described in step S4 includes: vectorizing the knowledge units constructed from the fused relay protection model; converting the knowledge units into 128-dimensional semantic vectors using the Sentence-BERT model; transforming discrete knowledge units into high-dimensional semantic vectors; and achieving multidimensional data similarity matching by calculating the distance between different knowledge units in the semantic space. The semantic distance is calculated using Euclidean distance, as shown in the following formula. Formula 8 In Equation 8, u , v A semantic vector representing two knowledge units; , Describe the first of two vectors i One component; n Indicates the dimension of the vector; By calculating semantic distance, knowledge units with a semantic distance less than a threshold of 0.5 are associated to obtain semantic information. Using the "entity-relationship-entity" triple as the core, the extracted text-based knowledge and image-based knowledge are topologically linked to construct a dynamic semantic association network; After the comprehensive multimodal knowledge graph is generated, all knowledge elements in the dynamic semantic association network are integrated to generate a comprehensive multimodal knowledge graph for relay protection. The knowledge graph consists of an entity layer, a relationship layer, and an event layer: the entity layer includes equipment entities, operation entities, risk entities, and terminology entities; the relationship layer includes the relationships between equipment, between equipment and operation, between operation and risk, and between graph elements; the event layer includes equipment operation events, risk identification events, and safety measure implementation events.

Images