Multi-source heterogeneous data automatic mapping method and device based on semantic alignment

Abstract

The invention relates to a multi-source heterogeneous data automatic mapping method and device based on semantic alignment, and the method comprises the steps: determining a power entity and an entity feature vector according to a power domain ontology model, and determining a power entity mapped by a to-be-aligned field based on the similarity between the entity feature vector and a feature vector of the to-be-aligned field; updating a message transmission mechanism by taking the constructed entity constraint condition as a constraint condition of a message function; inputting a full connection graph taking the to-be-aligned fields as nodes into the graph neural network, and obtaining an embedded vector of each node based on the updated message passing mechanism; when a candidate mapping relation is determined according to the semantic similarity of the two embedded vectors, entity category conflict detection and field conflict detection are carried out, and the candidate mapping relation is corrected according to a detection result; and generating data in a target format based on the corrected candidate mapping relationship. According to the method and the device, sharing, management and compatibility of multi-source heterogeneous data in different systems are realized.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method and apparatus for automatic mapping of multi-source heterogeneous data based on semantic alignment. Background Technology

[0002] In the power industry, different power business systems are responsible for different power businesses. For example, different business data such as standard clauses, power equipment parameters, and technical specifications reside in different business systems. There are differences in the semantic expression of the same data, such as field naming, data format, and units of measurement, between different power business systems. This results in multi-source heterogeneous data. When using data from different systems to process power tasks, it is difficult to achieve data sharing, semantic alignment, and data fusion for multi-source heterogeneous data. Furthermore, many power standard clauses exist in text-based natural language, which computers struggle to directly understand, lacking a unified semantic expression method, making it difficult to integrate and automatically map multi-source heterogeneous data.

[0003] In existing technologies, data integration and mapping are typically carried out by manually defining field mapping rules or using simple string matching algorithms. Manually defining field mapping rules cannot cover massive and dynamic professional expressions, while simple string matching algorithms, based on surface-level character similarity, cannot cover complex semantic structures and implicit semantics of text. This leads to semantic mapping errors at the field level for multi-source heterogeneous data, resulting in unit conflicts, duplicate definitions, and logical contradictions after data merging. Therefore, multi-source heterogeneous data between different power business systems faces difficulties in sharing, compatibility, and management. Summary of the Invention

[0004] This application provides a method and apparatus for automatic mapping of multi-source heterogeneous data based on semantic alignment, so as to at least solve the problem of difficulty in sharing, compatibility and management of multi-source heterogeneous data in related technologies.

[0005] In a first aspect, embodiments of this application provide an automatic mapping method for multi-source heterogeneous data based on semantic alignment, including: A power domain ontology model is constructed based on the meta-dataset and power domain constraint rules. The power domain ontology model is used to determine the set of power entities and the entity feature vector of each power entity. Based on the similarity between the entity feature vector and the feature vector of the field to be aligned, the power entity mapped to each field to be aligned is determined. Based on the mapping consistency between fields and the power entity, the semantic relationship between entities, and the path similarity of the power entities, entity constraints are constructed. The entity constraints are used as constraints for message functions. The constrained message functions, aggregation functions, and update functions are used to construct a message passing mechanism. Using the fields to be aligned as nodes, a fully connected graph is constructed with the similarity between fields as edge weights. The fully connected graph is then input into a graph neural network, which iteratively updates the node vector of each node through the message passing mechanism to obtain the embedding vector of each node. When the semantic similarity of the embedding vectors of any two nodes is greater than a preset threshold, the two fields to be aligned corresponding to these two nodes are used to form a candidate mapping relationship. Entity category conflict detection and field conflict detection are performed on each candidate mapping relationship, and the candidate mapping relationship is corrected according to the detection results. Based on the corrected candidate mapping relationship, the data contained in the field to be aligned is processed to generate data in the target format.

[0006] In one embodiment, after performing entity category conflict detection and field conflict detection on each of the candidate mapping relationships, the method further includes: A semantic link is generated based on the field to be aligned, the semantic vector of the field to be aligned, the propagation weights of the graph neural network, the power entity, and the mapping result; Based on the detection results, the conflict nodes of the semantic link are determined and the confidence level of the conflict nodes is obtained; When the confidence level is greater than or equal to the first preset threshold, the conflict content to be corrected is determined in the conflict node according to the conflict type.

[0007] In one embodiment, the entity constraint information includes entity category consistency constraints, semantic relationship constraints, and ontology path similarity constraints. The step of constructing entity constraint conditions based on the mapping consistency between fields and the power entity, entity semantic relationships, and the path similarity of the power entity includes: The entity category consistency constraint is constructed based on whether the power entities mapped by any two nodes are the same. The semantic relationship constraints are constructed based on the semantic relationship between the power entities mapped by any two nodes; The ontology path similarity constraint is constructed based on the path similarity between any two nodes mapping the power entity.

[0008] In one embodiment, the step of constructing a message passing mechanism using the entity constraints as constraints for the message functions, and the constrained message functions, aggregation functions, and update functions, includes: Based on the entity category consistency constraint, the original vector of the node is obtained. If any node and its neighboring node are mapped to the same power entity, a non-zero original vector is obtained between the node and its neighboring node. If any node and its neighboring node are mapped to mutually exclusive power entities, the original vector between the node and its neighboring node is a zero vector. Based on the semantic relationship constraints, virtual edges of the node are generated, and the neighborhood of the node is updated based on the edges of the node and the virtual edges; Based on the ontology path similarity constraint, the similarity constraint is used as a bias term, and the attention weight of the message function is updated based on the bias term; For each node, the node's aggregated representation is obtained by aggregating all its original vectors using the aggregation function based on the updated neighborhood. Based on the aggregated representation and the current feature vector of the node, the node's embedding vector is updated using the update function.

[0009] In one embodiment, determining the set of power entities and the entity feature vector of each power entity includes: Based on the power domain ontology model, several power entities and the feature information of each power entity are determined. The feature information is vectorized using a word vector model to obtain the entity feature vector for each power entity.

[0010] In one embodiment, after correcting the candidate mapping relationship based on the detection results, the method further includes: Based on the semantic similarity, the association weights of the nodes to be aligned in the graph neural network, and the field matching conditions, the comprehensive confidence of each corrected candidate mapping relationship is determined. When the overall confidence level is less than a preset confidence threshold, the candidate mapping relationship corresponding to the overall confidence level is designated as the object to be corrected. The entity category conflict detection and field conflict detection are performed on the object to be corrected. When a conflict is detected, the object to be corrected is corrected, and the corrected object to be corrected is fed back to the mapping rule base and semantic model.

[0011] In one embodiment, processing the data contained in the field to be aligned based on the corrected candidate mapping relationship to generate data in the target format includes: The structure of the output data is determined based on the revised candidate mapping relationship; Based on the structure, the data contained in the field to be aligned is reconstructed, and the reconstructed data is used to generate data in the target format according to the structure.

[0012] In one embodiment, the edge weights are specifically configured as follows: Obtain the feature vector of each node, and obtain the cosine similarity and Jaccard similarity of any two feature vectors; The edit distance is determined based on the text encoding of the two feature vectors. Based on the power entities mapped by the nodes, the enhanced similarity is determined according to the vector similarity of the entity feature vectors of the two power entities and the domain dictionary; The edge weights are determined based on the cosine similarity, the Jaccard similarity, the edit distance, the enhanced similarity, and the preset weights.

[0013] Secondly, embodiments of this application provide an automatic mapping device for multi-source heterogeneous data based on semantic alignment, comprising: The power entity determination module is used to construct a power domain ontology model based on a metadata dataset and power domain constraint rules, determine a set of power entities and the entity feature vector of each power entity through the power domain ontology model, and determine the power entity mapped to each of the fields to be aligned based on the similarity between the entity feature vector and the feature vector of the field to be aligned. The message passing mechanism update module is used to construct entity constraints based on the mapping consistency between fields and the power entity, the semantic relationship between entities, and the path similarity of the power entity. The entity constraints are used as constraints for message functions, and the constrained message functions, aggregation functions, and update functions are used to construct the message passing mechanism. The graph neural network processing module is used to construct a fully connected graph with the similarity between fields as the edge weights, input the fully connected graph into the graph neural network, and the graph neural network iteratively updates the node vector of each node through the message passing mechanism to obtain the embedding vector of each node. The correction module is used to form a candidate mapping relationship by combining the two fields to be aligned corresponding to the two nodes when the semantic similarity of the embedding vectors of any two nodes is greater than a preset threshold, perform entity category conflict detection and field conflict detection on each candidate mapping relationship, and correct the candidate mapping relationship according to the detection results. The generation module is used to process the data contained in the field to be aligned based on the corrected candidate mapping relationship and generate data in the target format.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the automatic mapping method for multi-source heterogeneous data based on semantic alignment as described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic alignment-based automatic mapping method for multi-source heterogeneous data as described in the first aspect above.

[0016] The method and apparatus for automatic mapping of multi-source heterogeneous data based on semantic alignment provided in this application have at least the following technical effects: By constructing entity constraints to constrain message functions in the message passing mechanism, the mechanism is updated. A fully connected graph with fields as nodes establishes deep semantic relationships between fields based on the association between fields and power entities during the update process. This allows the obtained embedding vectors to effectively distinguish between synonymous and heteronymous fields, achieving accurate semantic alignment at the field level. Candidate mapping relationships are determined based on the semantic similarity of any two embedding vectors. Within each candidate mapping relationship, entity category conflict detection and field conflict detection are used. Entity category conflict detection ensures the mapping relationship between each field to be aligned and the power entity, preventing cross-semantic errors. Field conflict detection ensures the consistency of attributes and structure of the fields to be aligned, guaranteeing semantic compliance and avoiding semantic misjudgments. The candidate mapping relationships are then corrected to achieve accurate automatic mapping of the fields to be aligned and output data in the target format, usable by different power systems. This solves the problem of difficulty in sharing, compatibility, and management of multi-source heterogeneous data between different power business systems.

[0017] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating automatic mapping of multi-source heterogeneous data based on semantic alignment, according to an exemplary embodiment. Figure 2 This is a schematic diagram of an automatic mapping device for multi-source heterogeneous data based on semantic alignment, according to an exemplary embodiment. Figure 3 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0020] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0022] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] Firstly, embodiments of this application provide an automatic mapping method for multi-source heterogeneous data based on semantic alignment. Figure 1 This is a flowchart illustrating automatic mapping of multi-source heterogeneous data based on semantic alignment, according to an exemplary embodiment, such as... Figure 1 As shown, the automatic mapping method for multi-source heterogeneous data based on semantic alignment includes: Step S101: Construct an ontology model for the power domain based on the meta-dataset and power domain constraint rules. Determine the set of power entities and the entity feature vector of each power entity through the ontology model. Based on the similarity between the entity feature vector and the feature vector of the field to be aligned, determine the power entity mapped to each field to be aligned.

[0024] Acquire multi-source heterogeneous data from different power business systems. This multi-source heterogeneous data includes metadata datasets and fields to be aligned. The metadata datasets include relational database data, hierarchical structured data, unstructured text data, and real-time data from different power business systems. Fields to be aligned are data fields set for the same type of data in different power business systems. Construct a power domain ontology model based on the metadata datasets and power domain constraint rules. Use this ontology model to determine the set of power entities and the entity feature vectors of each power entity. Both the entity feature vectors of the power entities and the feature vectors of the fields to be aligned are obtained using a word vector model. Obtain the similarity between the entity feature vectors and the feature vectors of the fields to be aligned. When the similarity exceeds a preset similarity threshold, determine the mapping relationship between the power entities and the fields to be aligned, establishing a mapping table between the fields to be aligned and the power entities.

[0025] The specific steps for determining the set of power entities and entity feature vectors through the power domain ontology model include: Based on the ontology model of the power domain, several power entities and the feature information of each power entity are determined.

[0026] The power domain ontology model includes objects that carry key business semantics and frequently appear in multi-source data as power entities. Each power entity has a unique identifier. For example, power entities include power standards, power terms, power indicators, power equipment, operating scenarios, power parameters, units, and safety levels. Several power entities are identified in the metadata set through the power domain ontology model.

[0027] By vectorizing feature information using word vector models, we obtain the entity feature vector for each power entity.

[0028] Based on power sector constraint rules and power entities, the constraint relationships between each power entity are determined, and the characteristic information of the power entities is obtained. This characteristic information includes the power entity's attribute information, unique identifier, and association information with other power entities, including dependency relationships, mutual exclusion constraints, and dependence relationships. The characteristic information is vectorized using a word vector model to obtain the entity feature vector of each power entity. The entity feature vector includes the entity's semantic information. Cosine clustering is performed on all entity feature vectors to determine the unique semantic meaning of each power entity. For example, semantic information such as rated capacity, Capacity, and CapRated are merged into synonymous concepts; semantic information such as equipment category, equipment type, and Type are grouped into the same semantic entity, so that each power entity has only one unique referential meaning in the system.

[0029] Based on the word vector model, the feature vector of each field to be aligned is obtained by vectorizing the field to be aligned. Since the vectorization model is the same word vector model, the feature vector of the field to be aligned and the entity feature vector are located in the same semantic space. The similarity between the feature vector and the entity feature vector is obtained. The field-electricity entity mapping relationship is established between the field to be aligned mapped by the feature vector with a similarity greater than a preset similarity threshold and the electricity entity mapped by the entity feature vector, so as to construct a mapping table that maps the field to be aligned to the electricity entity one by one.

[0030] Continuing with step S101, an ontology model for the power sector is constructed based on the metadata dataset in the multi-source heterogeneous data of the preset second format, determining each power entity and its entity feature vector. The power entity mapped to each field to be aligned is then determined based on the similarity between the feature vector of the field to be aligned and the entity feature vector, for use in subsequent steps for automatic mapping of the field to be aligned.

[0031] Step S102: Construct entity constraints based on the mapping consistency between fields and power entities, entity semantic relationships, and path similarity of power entities. Use entity constraints as constraints for message functions, and construct a message passing mechanism using constrained message functions, aggregation functions, and update functions.

[0032] Entity constraints include entity category consistency constraints, semantic relationship constraints, and ontology path similarity constraints. Specifically, entity category consistency constraints are constructed based on whether the power entities mapped to any two nodes are the same. Semantic relationship constraints are constructed based on the semantic relationships between the power entities mapped to any two nodes. Ontology path similarity constraints are constructed based on the path similarity between the power entities mapped to any two nodes.

[0033] Entity constraint bars serve as constraints on message functions in the message passing mechanism. By adding constraints on power entities during node updates, semantic alignment between different nodes can be achieved more accurately.

[0034] Continuing with step S102, entity category consistency constraints, semantic relationship constraints, and ontology path similarity constraints are used as constraints for the message function to implement the update of the message passing mechanism during the message passing process. Specifically, this includes: Based on entity category consistency constraints, if any node and its neighbor are mapped to the same power entity, a non-zero original vector is obtained between the node and its neighbor. This original vector has a coefficient greater than 1, increasing its size and giving it greater influence in subsequent aggregation stages, thus accelerating semantic alignment of similar fields. If any node and its neighbor are mapped to mutually exclusive power entities, the original vector between them is zero. Mutually exclusive power entities are those that cannot coexist in the same instance; for example, current and voltage are mutually exclusive power entities. When any two nodes are mapped to mutually exclusive power entities, it indicates a significant semantic gap between them. By setting the original vector to zero, the connection between the two nodes is severed, ensuring no informational link or mutual influence exists between them. Furthermore, this ensures that the number of neighbors remains constant in subsequent clustering stages, guaranteeing the stability of the graph data.

[0035] Based on semantic relation constraints, virtual edges of nodes are generated, and the neighborhood of nodes is updated based on the edges of nodes and virtual edges.

[0036] Semantic relationship constraints include the semantic relationships between the power entities mapped by any two nodes, specifically including subordination, dependency, and containment relationships. If the semantic relationships between two nodes satisfy subordination, dependency, and containment relationships, it indicates that the two nodes are semantically connected, and a virtual edge is established between these two nodes. A node has edges with its different neighboring nodes. Based on the original edges, the initial neighborhood can be determined. After adding virtual edges, previously unconnected nodes become neighbors, thus expanding the neighborhood range of each node. Therefore, the original edges and virtual edges determine a larger neighborhood for each node.

[0037] Based on ontology path similarity constraints, the similarity constraints are used as bias terms, and the attention weights of the message function are updated based on these bias terms.

[0038] An ontology path refers to the sequence of edges traversed when two power entities are connected in an ontology knowledge graph. The similarity of the mapping power entity paths of any two nodes is determined. Based on the ontology path similarity constraint, the similarity is used as a bias term for the node. The attention weight of the message function is updated based on the bias term so that during the aggregation phase, the node is made closer to neighboring nodes with the same power entity according to the updated attention weight.

[0039] For each node, an aggregated representation of the node is obtained by aggregating the original vector of the node using an aggregation function based on the updated neighborhood. Based on the aggregated representation and the feature vector of the current node, the node's embedding vector is updated using an update function. It should be noted that the aggregation function in this application employs conventional aggregation operations in the art, such as summation aggregation or mean aggregation. The update function uses a multilayer perceptron to perform a nonlinear transformation on the aggregation result; for example, the update function includes a two-layer fully connected network with ReLU activation.

[0040] The entity constraint restricts the message function, allowing the aggregation phase to obtain the aggregated representation of a node by aggregating its original vectors using the updated neighbors and attention weights. After aggregation, the node's embedding vector is updated using the update function based on the aggregated representation and the current node's feature vector, thus completing the update of the message passing mechanism.

[0041] Continuing with step S102, the message function is constrained based on entity constraints, and the original vector, node neighborhood, and attention weight in the message passing process are changed. Nodes are then clustered using a clustering function, and the node embedding vector is updated using an update function, thereby updating the message passing mechanism. The updated message passing mechanism allows for further consideration of the semantic relationship between power entities and nodes during node updates, effectively distinguishing between semantically similar but different names and suppressing erroneous mappings of similar but different names, thus improving the accuracy of semantic alignment of nodes.

[0042] Step S103: Using the fields to be aligned as nodes, construct a fully connected graph with the similarity between fields as edge weights. Input the fully connected graph into the graph neural network. The graph neural network iteratively updates the node vector of each node through a message passing mechanism to obtain the embedding vector of each node.

[0043] Using the fields to be aligned as nodes, an edge exists between any two nodes, and the similarity between the two nodes is used as the edge weight to construct a weighted fully connected graph. The initial feature vector of each node includes the field name, field description, data type, and contextual semantics. The fully connected graph is then input into a graph neural network, and based on the updated message passing mechanism, the embedding vector of each node is obtained.

[0044] The edge weights are specifically configured as follows: For each node, obtain its feature vector, and obtain the cosine similarity and Jaccard similarity between any two feature vectors; determine the edit distance based on the text encoding of the two feature vectors; based on the power entity mapped to the node, determine the enhanced similarity according to the similarity of the entity feature vectors of the two power entities and the domain dictionary; determine the edge weights based on the cosine similarity, the Jaccard similarity, the edit distance, and preset weights. The preset weights are weights pre-determined based on training data using a neural network. Semantic similarity satisfies: Among them, S core It is semantic similarity, S cos It is the cosine similarity, S jac It is the Jaccard similarity, S edit Edit distance, S domain It enhances similarity, and α, β, γ, and δ are preset weights.

[0045] By processing fully connected graphs through graph neural networks and an updated message passing mechanism, deep semantic associations between different fields to be aligned are realized. Furthermore, semantic alignment of fields with the same name but different meanings can be achieved based on the embedding vector, while avoiding semantic alignment of fields with the same name but different meanings.

[0046] Step S104: When the semantic similarity of the embedding vectors of any two nodes is greater than a preset threshold, the two fields to be aligned corresponding to these two nodes are combined into a candidate mapping relationship. Entity category conflict detection and field conflict detection are performed on each candidate mapping relationship, and the candidate mapping relationship is corrected according to the detection results.

[0047] When the semantic similarity between any two embedded vectors exceeds a preset threshold, the two fields to be aligned are combined into a candidate mapping relationship. For any candidate mapping relationship, entity category conflict detection and field conflict detection are performed. Entity category conflict detection checks whether the field to be aligned points to the corresponding power entity category. For example, if the field to be aligned is a capacity parameter and points to a pressure parameter entity category, then there is an entity category conflict. Field conflict detection includes field attribute conflict detection and field structure dependency conflict detection. Field attribute conflict detection checks whether the units, values, and ranges of the fields to be aligned are consistent; if they are inconsistent, a conflict occurs. Field structure dependency conflict detection checks whether the structural hierarchy of the context of the fields to be aligned is consistent.

[0048] If all conflict detection results are consistent, the current candidate mapping relationship will be retained. If there are inconsistencies in the conflict detection results, the candidate mapping relationship needs to be corrected to ensure its accuracy. The correction steps include: A semantic link is generated based on the field to be aligned, the semantic vector of the field to be aligned, the propagation weights of the graph neural network, the power entity, and the mapping result. For example, the semantic link satisfies: original field -> semantic vector -> graph neural network hierarchical propagation -> power entity -> mapping result. Through the semantic link, a visual and interpretable link can be formed for the candidate mapping relationship, revealing why the current candidate mapping relationship exists. The link provides semantic evidence for the field (text similarity, context similarity), ontology enhancement evidence (output weights of the graph neural network), structural evidence (field path, dependency structure), and decision rationale, including why the candidate mapping was selected and why conflicting mappings were excluded.

[0049] Based on the detection results, conflict nodes in the semantic link are identified, and the confidence level of each conflict node is obtained. When inconsistencies exist in the detection results, conflict nodes are identified in the semantic link according to the inconsistencies, along with their confidence levels. The confidence level characterizes the degree of credibility of the current conflict. A higher confidence level indicates a more accurate conflict.

[0050] When the confidence level is greater than or equal to the first preset threshold, the conflict content of the candidate mapping relationship is corrected in the conflict node according to the conflict type.

[0051] When the confidence level is greater than or equal to the first preset threshold, the conflict type is determined in the conflict node, and the conflict content of the candidate mapping relationship is corrected based on the conflict type. For example, if the conflict type is entity category conflict, and the field to be aligned does not point to the corresponding power entity category, it indicates that a serious mapping error has occurred, and the current erroneous candidate mapping relationship is deleted. If the conflict type is field attribute conflict, the inconsistent attribute information in the field to be aligned is corrected to align the attribute information.

[0052] Continuing with step S104, conflict detection is performed on the candidate mapping relationships, and semantic links are generated. These semantic links establish the traceability, interpretability, and verifiability of the candidate mapping relationships. When a conflict node is detected, it can be traced back to the conflict node according to the semantic links, and the candidate mapping relationship can be corrected, ensuring the accuracy of the field mapping relationships to be aligned. Furthermore, the semantic links also provide a visual perspective, enabling the determination of the content contained in each candidate mapping relationship for worker inspection.

[0053] Step S105: Based on the corrected candidate mapping relationship, process the data contained in the field to be aligned and generate data in the target format.

[0054] Based on the corrected candidate mapping relationship, the structure of the output data is determined, including standard field names, field semantic labels (unique identifiers of power entities), data types and unit specifications, field constraints (enumeration, range, dependency relationships), and structured organization methods.

[0055] The data contained in the field to be aligned is reconstructed based on the structure, and the reconstructed data is then used to generate data in the target format according to the structure.

[0056] Reconstruct the data contained in the fields to be aligned according to the structure of the output data, such as renaming the fields to be aligned to convert them into standard fields; converting the data units to a unified measurement system (SI units); performing field completion and generating missing field content according to inference rules; and reconstructing the nested structure of the data according to the standard data model. Based on the reconstructed data, output the data according to the target format, where the target format is a unified field order, hierarchical structure, and identifier naming convention.

[0057] Furthermore, to ensure the availability of the output data across different power business systems, different file formats are generated based on the output data to adapt to various power business systems. These file formats include JSON-LD, RDF / XML, and CSV / Parquet table formats.

[0058] Referring to step S105, in the corrected candidate mapping relationship, the data contained in the field to be aligned is reconstructed so that the data conforms to the structure of the output data, and data in the target format and different file formats are generated, thereby realizing data sharing and use between different business systems.

[0059] Continuing with step S101, both the metadata dataset and the fields to be aligned are multi-source heterogeneous data. Multi-source heterogeneous data originates from different power business systems. Before obtaining multi-source heterogeneous data, the original data from these different power business systems needs to be processed. The specific steps for obtaining multi-source heterogeneous data include the following: Step S111: Obtain raw data from different power systems, parse and structure the raw data to obtain data to be processed in a preset first format.

[0060] Raw data from different power systems is acquired, and different parsing and structuring processes are performed based on the data type of the raw data. The raw data includes relational database data, hierarchical structured data, unstructured text data, and real-time data.

[0061] For relational database data, a standard API protocol is used to connect to the power business system. Data obtained from the power business system is extracted at the field level, and the extracted data is updated based on the primary key, timestamp, and log offset using an incremental synchronization mechanism. The extracted data undergoes field normalization processing, which involves sequentially processing the fields through "field name standardization," "data type conversion," and "encoding format normalization" to obtain a unified and standardized field representation.

[0062] For hierarchical data, which is organized in a nested tree structure where parent nodes contain child nodes, and child nodes contain even deeper child nodes, forming a multi-level hierarchical relationship, the parser traverses all nodes, transforming the multi-level nested paths into flattened paths. Based on a knowledge pattern matching algorithm for the power industry, it determines the node types and field meanings in the flattened paths and maps these node types and field meanings to a preset format.

[0063] For unstructured text data, OCR is used to identify text content, syntactic dependency parsing is used to accurately capture the phrase structure of the text content, and a pre-trained language model based on the Transformer encoder architecture is used to identify key information in the text content. The key information includes clause titles, indicators, technical requirements, parameters and units, and the key information is encoded into triplet format.

[0064] For real-time data streams, real-time data is accessed through a streaming acquisition protocol, and a dynamic window mechanism is used to perform real-time aggregation, anomaly detection, and format unification of the real-time streaming data.

[0065] Continuing with step S111, the data is parsed and structured for different data types, and the processed data is output according to the preset first format to obtain the data to be processed.

[0066] Step S112: Detect abnormal data by detecting the data to be processed, and correct the abnormal data to obtain intermediate data.

[0067] Abnormal data includes invalid fields and erroneous data. Invalid fields in the data to be processed are identified, including null values, outliers, and illegal characters. Errors such as inconsistent units and encoding conflicts are identified using a field rule base. Invalid fields, erroneous data, and duplicate data are removed to ensure data accuracy. For text data, a duplicate detection threshold is established. If the number of occurrences of the same text data exceeds the threshold, the text data is considered duplicate data, thereby improving the accuracy of text data processing.

[0068] In addition, missing data is imputed. If the missing data is numerical, the mean, median, and KNN are calculated using data of the same type from historical time periods to fill in the missing numerical data. If the missing data is default value data, it is retrieved from a power industry dictionary and the corresponding data is filled in. If the missing data is time series data, it is filled in using predicted time data obtained from historical time series data through an RNN algorithm. It should be noted that the confidence level of each imputed data is calculated. If the confidence level is lower than a preset confidence threshold, it is marked as "requires manual verification" to ensure the accuracy of the data.

[0069] By identifying and removing abnormal data in the data to be processed, and filling in the abnormal data, the accuracy of the data is ensured.

[0070] Step S113: Organize and output the intermediate data according to the preset second format to obtain multi-source heterogeneous data.

[0071] The preset second format includes standard fields, field semantic labels, an initial field mapping rule set, data quality labels, and a standardized sample data document. Intermediate data is organized and output according to this preset second format to obtain multi-source heterogeneous data.

[0072] By processing the raw data through steps S111 to S113, multi-source heterogeneous data in a preset second format is obtained, which can provide accurate data for automatic mapping and lay the foundation for improving the accuracy of automatic mapping.

[0073] Following step S101, the automatic mapping method for multi-source heterogeneous data based on semantic alignment further includes storing the data. Specifically, the multi-source heterogeneous data is stored in a standard format to improve the efficiency and quality of data integration. Relationships between power entities are stored as triples.

[0074] In addition, a field mapping index was constructed, including a mapping table from fields to power entities, an index library from fields to synonym sets, and a directional mapping table from fields to data source systems. Based on the field mapping index, a unified data access interface was set up, including a standard semantic interface for corresponding fields, a multi-source field alignment query interface, and an alignment confidence feedback interface, so that information can be obtained through the interface.

[0075] Following step S104, the modified candidate mapping relationship is further adaptively corrected to achieve dynamic correction of the mapping relationship, specifically including: Based on semantic similarity, the association weights of the nodes to be aligned in the graph neural network, and field matching conditions, the comprehensive confidence of each corrected candidate mapping relationship is determined. The comprehensive confidence includes the semantic similarity in the candidate mapping relationship, the association weights between output nodes in the graph neural network, the matching conditions for field contextual consistency and structural dependencies, and the prior confidence based on historical mappings.

[0076] When the overall confidence level is less than the preset confidence threshold, the candidate mapping relationship corresponding to the overall confidence level is the object to be corrected.

[0077] The overall confidence level is compared with a preset confidence threshold, which includes a first confidence threshold and a second confidence threshold. Overall confidence levels below the first confidence threshold are considered anomalous mappings; those greater than or equal to the first confidence threshold but less than the second confidence threshold are considered low-confidence mappings; and those greater than or equal to the second confidence threshold are considered high-confidence mappings. The candidate mapping relationships corresponding to the anomalous mappings are then considered as objects to be corrected.

[0078] The system performs entity category conflict detection and field conflict detection on the object to be corrected. When a conflict is detected, the object to be corrected is corrected, and the corrected object is fed back to the mapping rule base and semantic model.

[0079] The system performs entity category conflict detection and field conflict detection on the objects to be corrected. When a conflict is detected, the objects are corrected. Correction measures include: deleting high-confidence conflicting mappings, replacing conflicting mappings with candidate mappings with higher semantic consistency, reducing the confidence of potentially conflicting mappings for subsequent decision-making, and performing manual verification when the mapping conflict intensity reaches a warning threshold. The corrected objects are fed back into the mapping rule base and semantic model for parameter updates and optimization in subsequent mapping tasks. A mapping correction log is recorded to form a traceable mapping evolution record to support the long-term operation of automatic mapping.

[0080] In summary, the automatic mapping method for multi-source heterogeneous data based on semantic alignment provided in this application constructs entity constraints to constrain message functions in the message passing mechanism, thereby updating the message passing mechanism. The fully connected graph with fields as nodes can establish deep semantic relationships between fields based on the association between fields and power entities during the update process. This allows the obtained embedding vectors to effectively distinguish between fields with different names but the same meaning and fields with different names but different meanings, achieving accurate semantic alignment at the field level. Candidate mapping relationships are determined based on the semantic similarity of any two embedding vectors. In each candidate mapping relationship, entity category conflict detection and field conflict detection are used. Entity category conflict detection ensures the mapping relationship between each field to be aligned and the power entity, preventing cross-semantic errors in mapping. Field conflict detection ensures the consistency of attributes and structure of the fields to be aligned, guaranteeing semantic compliance and avoiding semantic misjudgment. The candidate mapping relationships are then corrected to achieve accurate automatic mapping of the fields to be aligned and output data in the target format, which can be used by different power systems. This solves the problem of difficulty in sharing, compatibility, and management of multi-source heterogeneous data between different power business systems.

[0081] Secondly, embodiments of this application provide an automatic mapping device for multi-source heterogeneous data based on semantic alignment. Figure 2 This is a schematic diagram of an automatic mapping device for multi-source heterogeneous data based on semantic alignment, according to an exemplary embodiment. Figure 2 As shown, the automatic mapping device for multi-source heterogeneous data based on semantic alignment includes: The power entity determination module is used to construct a power domain ontology model based on the meta-dataset and power domain constraint rules, determine the set of power entities and the entity feature vector of each power entity, and determine the power entity mapped to each field to be aligned based on the similarity between the entity feature vector and the feature vector of the field to be aligned. The message passing mechanism update module is used to construct entity constraints based on the mapping consistency between fields and power entities, entity semantic relationships, and path similarity of power entities. The entity constraints are then used as constraints for message functions to update the message passing mechanism. The graph neural network processing module is used to construct a fully connected graph with the fields to be aligned as nodes and the similarity between fields as edge weights. The fully connected graph is then input into the graph neural network, and the embedding vector of each node is obtained based on the updated message passing mechanism. The correction module is used to form a candidate mapping relationship between two fields to be aligned when the semantic similarity between any two embedded vectors is greater than a preset threshold. It performs entity category conflict detection and field conflict detection on each candidate mapping relationship and corrects the candidate mapping relationship based on the detection results. The generation module is used to process the data contained in the field to be aligned based on the corrected candidate mapping relationship and generate data in the target format.

[0082] In summary, the semantic alignment-based automatic mapping device for multi-source heterogeneous data provided in this application updates the message passing mechanism by constructing entity constraints to constrain message functions. The fully connected graph with fields as nodes establishes deep semantic relationships between fields based on the association between fields and power entities during the update process. This allows the obtained embedding vectors to effectively distinguish between synonymous and heteronymous fields, achieving accurate semantic alignment at the field level. Candidate mapping relationships are determined based on the semantic similarity of any two embedding vectors. Within each candidate mapping relationship, entity category conflict detection and field conflict detection are used. Entity category conflict detection ensures the mapping relationship between each field to be aligned and the power entity, preventing cross-semantic errors. Field conflict detection ensures the consistency of attributes and structure of the fields to be aligned, guaranteeing semantic compliance and avoiding semantic misjudgments. The candidate mapping relationships are then corrected to achieve accurate automatic mapping of the fields to be aligned and output data in the target format, usable by different power systems. This solves the problem of difficulty in sharing, compatibility, and management of multi-source heterogeneous data between different power business systems.

[0083] It should be noted that the semantically aligned multi-source heterogeneous data automatic mapping device provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," and "subunit" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0084] Thirdly, embodiments of this application provide an electronic device, Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 3 As shown, the electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0085] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0086] The memory 82 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include removable or non-removable (or fixed) media. Where appropriate, the memory 82 may be internal or external to a data processing device. In a particular embodiment, the memory 82 is non-volatile memory. In a particular embodiment, the memory 82 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0087] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0088] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any of the semantic alignment-based automatic mapping methods for multi-source heterogeneous data in the above embodiments.

[0089] In one embodiment, the semantically aligned multi-source heterogeneous data automatic mapping device may further include a communication interface 83 and a bus 80. Wherein, as Figure 3 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0090] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0091] Bus 80 includes hardware, software, or both, that couples the components of the foil capacitor damage condition assessment device together. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0092] Fourthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the foil capacitor damage state assessment method provided in the first aspect.

[0093] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0094] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to perform the steps of implementing the semantically aligned multi-source heterogeneous data automatic mapping method provided in the first aspect.

[0095] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0096] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for automatic mapping of multi-source heterogeneous data based on semantic alignment, characterized in that, include: A power domain ontology model is constructed based on the meta-dataset and power domain constraint rules. The power domain ontology model is used to determine the set of power entities and the entity feature vector of each power entity. Based on the similarity between the entity feature vector and the feature vector of the field to be aligned, the power entity mapped to each field to be aligned is determined. Based on the mapping consistency between fields and the power entity, the semantic relationship between entities, and the path similarity of the power entities, entity constraints are constructed. The entity constraints are used as constraints for message functions. The constrained message functions, aggregation functions, and update functions are used to construct a message passing mechanism. Using the fields to be aligned as nodes, a fully connected graph is constructed with the similarity between fields as edge weights. The fully connected graph is then input into a graph neural network, which iteratively updates the node vector of each node through the message passing mechanism to obtain the embedding vector of each node. When the semantic similarity of the embedding vectors of any two nodes is greater than a preset threshold, the two fields to be aligned corresponding to these two nodes are used to form a candidate mapping relationship. Entity category conflict detection and field conflict detection are performed on each candidate mapping relationship, and the candidate mapping relationship is corrected according to the detection results. Based on the corrected candidate mapping relationship, the data contained in the field to be aligned is processed to generate data in the target format.

2. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 1, characterized in that, After performing entity category conflict detection and field conflict detection on each of the candidate mapping relationships, the method further includes: A semantic link is generated based on the field to be aligned, the semantic vector of the field to be aligned, the propagation weights of the graph neural network, the power entity, and the mapping result; Based on the detection results, the conflict nodes of the semantic link are determined and the confidence level of the conflict nodes is obtained; When the confidence level is greater than or equal to the first preset threshold, the conflict content to be corrected is determined in the conflict node according to the conflict type.

3. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 1, characterized in that, The entity constraint information includes entity category consistency constraints, semantic relationship constraints, and ontology path similarity constraints. The construction of entity constraint conditions based on the mapping consistency between fields and the power entity, entity semantic relationships, and the path similarity of the power entity includes: The entity category consistency constraint is constructed based on whether the power entities mapped by any two nodes are the same. The semantic relationship constraints are constructed based on the semantic relationship between the power entities mapped by any two nodes; The ontology path similarity constraint is constructed based on the path similarity between any two nodes mapping the power entity.

4. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 3, characterized in that, The mechanism for constructing a message passing mechanism, which uses the entity constraints as constraints for the message functions and the constrained message functions, aggregation functions, and update functions, includes: Based on the entity category consistency constraint, the original vector of the node is obtained. If any node and its neighboring node are mapped to the same power entity, a non-zero original vector is obtained between the node and its neighboring node. If any node and its neighboring node are mapped to mutually exclusive power entities, the original vector between the node and its neighboring node is a zero vector. Based on the semantic relationship constraints, virtual edges of the node are generated, and the neighborhood of the node is updated based on the edges of the node and the virtual edges; Based on the ontology path similarity constraint, the similarity constraint is used as a bias term, and the attention weight of the message function is updated based on the bias term; For each node, the node's aggregated representation is obtained by aggregating all its original vectors using the aggregation function based on the updated neighborhood. Based on the aggregated representation and the current feature vector of the node, the node's embedding vector is updated using the update function.

5. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 1, characterized in that, The determination of the set of power entities and the entity feature vector of each power entity includes: Based on the power domain ontology model, several power entities and the feature information of each power entity are determined. The feature information is vectorized using a word vector model to obtain the entity feature vector for each power entity.

6. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 1, characterized in that, After correcting the candidate mapping relationship based on the detection results, the process also includes: Based on the semantic similarity, the association weights of the nodes to be aligned in the graph neural network, and the field matching conditions, the comprehensive confidence of each corrected candidate mapping relationship is determined. When the overall confidence level is less than a preset confidence threshold, the candidate mapping relationship corresponding to the overall confidence level is designated as the object to be corrected. The entity category conflict detection and field conflict detection are performed on the object to be corrected. When a conflict is detected, the object to be corrected is corrected, and the corrected object to be corrected is fed back to the mapping rule base and semantic model.

7. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 1, characterized in that, The process of processing the data contained in the field to be aligned based on the corrected candidate mapping relationship to generate data in the target format includes: The structure of the output data is determined based on the revised candidate mapping relationship; Based on the structure, the data contained in the field to be aligned is reconstructed, and the reconstructed data is used to generate data in the target format according to the structure.

8. The automatic mapping method for multi-source heterogeneous data based on semantic alignment according to claim 1, characterized in that, The specific configuration of the edge weights is as follows: Obtain the feature vector of each node, and obtain the cosine similarity and Jaccard similarity of any two feature vectors; The edit distance is determined based on the text encoding of the two feature vectors. Based on the power entities mapped by the nodes, the enhanced similarity is determined according to the vector similarity of the entity feature vectors of the two power entities and the domain dictionary; The edge weights are determined based on the cosine similarity, the Jaccard similarity, the edit distance, the enhanced similarity, and the preset weights.

9. An automatic mapping device for multi-source heterogeneous data based on semantic alignment, characterized in that, include: The power entity determination module is used to construct a power domain ontology model based on a metadata dataset and power domain constraint rules, determine a set of power entities and the entity feature vector of each power entity through the power domain ontology model, and determine the power entity mapped to each of the fields to be aligned based on the similarity between the entity feature vector and the feature vector of the field to be aligned. The message passing mechanism update module is used to construct entity constraints based on the mapping consistency between fields and the power entity, the semantic relationship between entities, and the path similarity of the power entity. The entity constraints are used as constraints for message functions, and the constrained message functions, aggregation functions, and update functions are used to construct the message passing mechanism. The graph neural network processing module is used to construct a fully connected graph with the similarity between fields as the edge weights, input the fully connected graph into the graph neural network, and the graph neural network iteratively updates the node vector of each node through the message passing mechanism to obtain the embedding vector of each node. The correction module is used to form a candidate mapping relationship by combining the two fields to be aligned corresponding to the two nodes when the semantic similarity of the embedding vectors of any two nodes is greater than a preset threshold, perform entity category conflict detection and field conflict detection on each candidate mapping relationship, and correct the candidate mapping relationship according to the detection results. The generation module is used to process the data contained in the field to be aligned based on the corrected candidate mapping relationship and generate data in the target format.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the semantically aligned automatic mapping method for multi-source heterogeneous data as described in any one of claims 1 to 8.

Images