Database fusion matching method and system based on multi-dimensional index correlation analysis

Abstract

The invention belongs to the field of electric power data fusion, and provides a database fusion matching method and system based on multi-dimensional index correlation analysis, and the technical scheme is as follows: obtaining different types of electric power databases and database tables; extracting metadata information in different types of power databases; according to the database table and the extracted metadata information in the different types of power databases, converting field information of the databases into graph node information, respectively constructing a table entity, an attribute entity and a table-attribute relationship, and generating a relationship graph; respectively extracting local features and global features in the relational graph through a graph convolutional network and a self-attention mechanism on the basis of node attributes in the relational graph and the relation between the nodes, and splicing the local features and the global features to obtain representation vectors of the nodes; based on the representation vectors of the nodes, the Jaccard similarity between each pair of nodes is calculated, the Jaccard similarity is adopted, field matching is achieved through node matching, fusion matching of the database is completed, and the field matching accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of electric power data fusion, and in particular relates to a database fusion matching method and system based on multi-dimensional indicator association analysis. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of cloud computing and big data, different cloud data management solutions such as NoSQL, distributed file systems (such as Hadoop HDFS) and big data processing frameworks (such as Hadoop MapReduc or Apache Spark) have emerged in large numbers, specifically targeting different types of data and being able to perform orders of magnitude better than traditional relational data. However, in practical applications, a single type of data storage method can no longer meet the needs of diversified data storage and query. Therefore, most companies have begun to deploy multiple storage systems to optimize operational loads and deploy multiple or even multiple types of databases to take advantage of the best features of each database.

[0004] Today, the data assets of power companies present typical big data characteristics. These power data come from all aspects of power generation and power use, including power generation, transmission, transformation, distribution and dispatching, including various types of data such as power grid operation, equipment management, marketing services and enterprise management, and contain rich information reflecting the production and operation and customer service status of power companies. After years of accumulation, the amount of enterprise data is huge; at the same time, in multi-level management and multi-professional control, there are problems such as diverse data calibers, differences in data calibers of various professions, and ambiguity in comprehensive data;

[0005] On the one hand, since the above data are scattered in different units and different professional application systems, it may lead to the formation of information islands, hindering information sharing and communication between different departments, and affecting the consistency and efficiency of cross-departmental collaboration and decision-making; on the other hand, it causes data fragmentation and dispersion, increases the complexity of data access and management, reduces data availability and accessibility, and data inconsistency and inaccuracy, bringing uncertainty and risks to corporate decision-making and business operations. Redundant storage and repeated collection of data increase data management and storage costs, and waste storage resources. Summary of the invention

[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a database fusion matching method and system based on multidimensional indicator association analysis, which takes into account the wide variety of metadata information of the power database, and considers multiple factors when performing similarity association matching. The database fusion matching method based on multidimensional indicator association analysis improves the field matching accuracy, establishes a unified data view, and helps developers to query different databases according to needs in different applications.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a database fusion matching method based on multi-dimensional indicator association analysis, comprising the following steps:

[0009] Get different types of power databases and database tables;

[0010] Extract metadata information from different types of power databases;

[0011] According to the database table and the metadata information extracted from different types of power databases, the field information of the database is converted into graph node information, and table entities, attribute entities and table-attribute relationships are constructed respectively to generate a relationship graph;

[0012] Based on the node attributes and the connections between nodes in the relationship graph, the local features and global features in the relationship graph are extracted through the graph convolution network and the self-attention mechanism. After splicing the local features and the global features, the representation vector of the node is obtained;

[0013] Based on the representation vector of the node, the Jaccard similarity between each pair of nodes is calculated. The Jaccard similarity is used to achieve field matching through node matching to complete the fusion matching of the database.

[0014] Furthermore, the different types of power databases include Oracle database, Mysql database, Mongodb database, HDFS database and Hive database;

[0015] The metadata includes field information, structural information, and data sample information; the field information includes name, annotation, structure, and constraints; the structural information includes the table to which it belongs and other fields and table information associated with the graph; and the data sample information includes the distribution, type, common features, and patterns of data values.

[0016] Furthermore, based on the node attributes in the relationship graph and the connections between the nodes, the local features and global features in the relationship graph are extracted through the graph convolution network and the self-attention mechanism, including:

[0017] Combine the node attributes and the connections between nodes in the relationship graph, and use the graph convolutional neural network to extract the local features of the nodes in the relationship graph;

[0018] After the output of each GCN layer in the graph convolutional neural network, a self-attention mechanism is introduced to learn the weights between nodes to capture the relationship between nodes, so as to further optimize the node representation and obtain global features; wherein, the introduction of the self-attention mechanism and the learning of the weights between nodes to capture the relationship between nodes include:

[0019] Calculate the self-attention weight, calculate the normalized attention weight, use the attention weight to aggregate neighbor information, update the node representation based on the vector after aggregating neighbor information, and obtain the global feature.

[0020] Furthermore, in the relationship graph, local features include: node degree, neighbor information and clustering coefficient, and global features include: graph density, average shortest path length, network diameter and network connectivity.

[0021] Furthermore, the node-based representation vectors are used to calculate the Jaccard similarity between each pair of nodes, and the Jaccard similarity is used to achieve field matching through node matching, including:

[0022] Calculate each node in different relationship graphs, calculate the set of neighbor nodes of the node, and obtain the set of nodes adjacent to the node;

[0023] The Jaccard similarity formula is used to calculate the similarity of nodes in any two different relationship graphs, and the nodes with the highest similarity are used as matching nodes until matching nodes are selected for the nodes in each relationship graph.

[0024] Furthermore, the method also includes: identifying similar or identical nodes in different databases, accurately matching and comparing the attributes and related information of successfully matched node pairs, and then establishing a new node based on the attributes and related information of these successfully matched nodes; for nodes that are not successfully matched, retaining their original node attributes and related information, these successfully matched new nodes and unmatched old nodes form a new relationship graph, establishing a new database based on the new relationship graph, and extracting the required data view based on the new database.

[0025] A second aspect of the present invention provides a database fusion matching system based on multi-dimensional indicator association analysis, comprising:

[0026] A database acquisition module, which is used to acquire different types of power databases and database tables;

[0027] A metadata information extraction module, which is used to extract metadata information from different types of power databases;

[0028] A relationship graph generation module is used to convert the field information of the database into graph node information according to the metadata information in the database table and the extracted different types of power databases, respectively construct table entities, attribute entities and table-attribute relationships, and generate a relationship graph;

[0029] The node representation module is used to extract local features and global features in the relationship graph based on the node attributes and the connections between nodes in the relationship graph through the graph convolution network and the self-attention mechanism, and then concatenate the local features and the global features to obtain the node representation vector;

[0030] The fusion matching module is used to calculate the Jaccard similarity between each pair of nodes based on the node representation vector, and adopts the Jaccard similarity to achieve field matching through node matching to complete the fusion matching of the database.

[0031] A third aspect of the present invention provides a computer-readable storage medium.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the database fusion matching method based on multi-dimensional indicator association analysis as described above.

[0033] A fourth aspect of the present invention provides a computer device.

[0034] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the database fusion matching method based on multi-dimensional indicator association analysis as described above are implemented.

[0035] A fifth aspect of the present invention provides a program product.

[0036] A program product, including the program product being a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps in the database fusion matching method based on multidimensional indicator association analysis as described above are implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. In view of the problem that data scattered in different units and systems may lead to the formation of information islands, hinder information sharing and communication between different departments, and affect the consistency and efficiency of cross-departmental collaboration and decision-making, the present invention proposes a database fusion matching method based on multi-dimensional indicator association analysis. Taking into account the wide variety of metadata information in the power database, the metadata of different databases, namely field information, structure information, and data sample information, are considered when performing similarity association analysis. By combining graph neural networks, node features are extracted, and databases are merged based on node features to solve problems such as information islands and communication barriers.

[0039] 2. In view of the fact that data is scattered in different units and different professional application systems, which causes data fragmentation and dispersion, increases the complexity of data access and management, and reduces the availability and accessibility of data, the present invention establishes a unified data view based on the fused and matched database, identifies similar or identical nodes in different databases, and accurately matches and compares the attributes and related information of the successfully matched node pairs, and then establishes a new node based on the attributes and related information of these successfully matched nodes; for the nodes that are not successfully matched, retain their original node attributes and related information, and these successfully matched new nodes and the unmatched old nodes form a new relationship graph, and a new database is established based on the new relationship graph. Then, the staff can use this new database to extract the data view required for the work. It can help developers to query different databases according to needs in different applications, reduce the learning cost of developers' multi-system databases, and improve the generalization of multi-storage systems.

[0040] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 It is an overall flow chart of a database fusion matching method based on multi-dimensional indicator association analysis provided by an embodiment of the present invention;

[0043] Figure 2 It is a schematic diagram of a database fusion matching structure based on multi-dimensional indicator association analysis provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] Explanation of terms

[0048] Multidimensional indicator association analysis: refers to considering the metadata of different databases when conducting similarity association analysis, namely field information (name, annotation, structure, constraints, etc.), structural information (belonging table and other fields and table information associated in the graph), data sample information (distribution, type, common characteristics and patterns of data values, etc.).

[0049] The database fusion matching method can use the relation graph node alignment. Relation graph node alignment is an important task in graph data processing, which aims to align or match nodes in different graphs to achieve cross-graph information transfer, knowledge fusion and cross-network data analysis.

[0050] The task of graph node alignment is to find similar or corresponding nodes in different graphs to establish associations between nodes. This usually involves node matching between two or more graphs.

[0051] Relationship graph node alignment has been widely used in many fields, including social network analysis, knowledge graph fusion, bioinformatics, recommendation systems, financial risk analysis, etc.

[0052] In social networks, it helps to discover correspondence between users on different social media platforms.

[0053] In the field of knowledge graphs, it can help align entities in different knowledge graphs to achieve knowledge fusion and cross-graph queries.

[0054] Methods for aligning nodes in relationship graphs include graph embedding-based techniques, graph neural networks (GNNs) methods, subgraph matching algorithms, and methods based on network isomorphism detection.

[0055] These methods usually use information such as node attributes, connection patterns, and graph structures to perform node matching.

[0056] In response to the problem mentioned in the background technology of this application that the data mentioned in different units and systems are scattered, which may lead to the formation of information islands, hinder information sharing and communication between different departments, and affect the consistency and efficiency of cross-departmental collaboration and decision-making, the present invention proposes a database fusion matching method based on multidimensional indicator association analysis. When performing similarity association analysis, the metadata of different databases, namely field information, structure information, and data sample information are considered. By combining graph neural networks, node features are extracted, and databases are merged based on node features to solve problems such as information islands and communication barriers. In view of the fact that the data mentioned in the background technology of this application are scattered in different units and different professional application systems, which causes data fragmentation and dispersion, increases the complexity of data access and management, and reduces the availability and accessibility of data, the present invention establishes a unified data view based on the fused and matched database, identifies similar or identical nodes in different databases, and accurately matches and compares the attributes and related information of the successfully matched node pairs, and then establishes a new node based on the attributes and related information of these successfully matched nodes; for the nodes that are not successfully matched, retain their original node attributes and related information, and these successfully matched new nodes and the unmatched old nodes form a new relationship graph, and a new database is established based on the new relationship graph. Then, the staff can use this new database to extract the data view required for the work. It can help developers to query different databases according to needs in different applications, reduce the learning cost of developers' multi-system databases, and improve the generalization of multi-storage systems.

[0057] Embodiment 1

[0058] like Figure 1 As shown, this embodiment provides a database fusion matching method based on multi-dimensional indicator association analysis, comprising the following steps:

[0059] Step 1, data acquisition: obtaining different types of power databases and database tables;

[0060] In this embodiment, various database data obtained based on the online platform are application examples;

[0061] Specifically, in step 1, the different types of power databases are distributed in different power systems, including Oracle database, Mysql database, Mongodb database, HDFS database and Hive database;

[0062] After acquiring different types of power databases, the data of each database is preprocessed, including data cleaning, missing value completion, data definition and storage.

[0063] Step 2, metadata extraction: metadata information from different types of power databases extracted according to database tables;

[0064] In this embodiment, in step 2, the metadata includes but is not limited to information about the field itself, such as name, annotation, structure, constraints, etc.; structural information: the table to which it belongs and other fields and table information associated with the graph; data sample information: distribution, type, common characteristics and patterns of data values, etc.

[0065] For example, in power data, power information (current, voltage, power consumption, etc.) is stored in cycles. In the voltage table, the metadata contains information about the voltage table field itself, such as field name (user ID, date, time, voltage data, etc.), field type (marking field type such as floating point type), field annotation (remark field information), etc.; structural information, such as the foreign key of the ID field in the voltage table points to the user ID in the user table; data sample information: for example, voltage field information distribution.

[0066] Step 3, relationship graph construction: According to the database table and the metadata information of different types of power databases extracted, the field information of the database is converted into graph node information, and table entities, attribute entities and table-attribute relationships are constructed respectively to generate a relationship graph;

[0067] In step 3, according to the database table and the metadata information in the different types of power databases extracted, the field information of the database is converted into graph node information, and table entities, attribute entities and table-attribute relationships are constructed respectively to generate a relationship graph:

[0068] For example, a table can be regarded as a node, such as a voltmeter. The fields in the table are node attributes, such as date_time, date_hour, three-phase voltage and other fields. The foreign key information between tables, such as the id in the voltmeter can correspond to the primary key CONS_id of the user table, can be regarded as the connection between nodes.

[0069] Step 4: Vectorized representation of graph nodes: Based on the node attributes and the connections between nodes in the relationship graph, the local features and global features in the relationship graph are extracted through the graph convolutional network and the self-attention mechanism. After concatenating the local features and the global features, the representation vector of the node is obtained.

[0070] In step 4, based on the node attributes in the relationship graph and the connections between nodes, the local features and global features in the relationship graph are extracted through the graph convolution network and the self-attention mechanism. After the local features and the global features are spliced, the representation vector of the node is obtained, which specifically includes:

[0071] Among them, the table is a node, the fields in the table are node attributes, and the foreign key information between tables can be regarded as the connection between nodes.

[0072] Based on the relationship graph G, the nodes are represented as V = [v0, v1, v2, ..., v N ], N is the total number of nodes. The adjacency matrix A, where A[i][j] represents the node v i To node v j The strength of the connection between them.

[0073] The graph convolutional network and self-attention mechanism are introduced to capture the local features and global features of the relationship graph nodes, respectively. Figure 2 shown.

[0074] Specifically, based on the node attributes in the relationship graph and the connections between the nodes, the local features and global features in the relationship graph are extracted through the graph convolution network and the self-attention mechanism, including the following steps:

[0075] Step 401: extracting local features of nodes in the relationship graph by combining node attributes and connections between nodes in the relationship graph through a graph convolutional neural network;

[0076] For each node v i There is an eigenvector represented by x i , where i ranges from 1 to N. These feature vectors may include attribute information or other features of the nodes.

[0077] Use GCN to aggregate the information of neighbor nodes and generate an embedded representation of the node. The output of the GCN layer can be expressed as:

[0078] H (l+1) =f(A·H (l) ·W (l) ) (1),

[0079] Among them, H (l) is the node representation matrix of the lth layer, H (0) =X, where X is the initial feature matrix, W (l) is the weight matrix of the lth layer, and f() is the ReLU activation function.

[0080] The GCN layer aggregates the neighbor node information of each node to generate local features of the node.

[0081] Step 402: After the output of each GCN layer, a self-attention mechanism is introduced to further optimize the node representation and obtain global features, such as domain name representation vectors. The goal of the self-attention mechanism is to learn the weights between nodes to better capture the relationship between nodes.

[0082] The introduction of the self-attention mechanism to further optimize the node representation includes the following steps:

[0083] Step 4021: Calculate self-attention weight:

[0084]

[0085] where e ij Represents node v i For node v j LeakReLU(·) is the activation function with leaky linear rectifier unit (Leaky ReLU), and Attention(·) is the self-attention function used to calculate the node v i For node v j The attention weight.

[0086] Step 4022: Calculate the normalized attention weights:

[0087]

[0088] where a ij Represents node v i For node v j The normalized attention weight of node v. i The set of neighbor nodes.

[0089] Step 4023: Aggregate neighbor information using attention weights:

[0090]

[0091] Step 4024: Update node representation:

[0092] H (2) =f(h′ i ·W (2) ) (5),

[0093] Among them, H (2) is the node representation matrix of the second GCN layer, that is, the global features obtained after the second aggregation, and its dimension is the same as H (1) Same. (2) is the weight matrix of the second GCN layer.

[0094] Step 4025: Repeat steps 4021 to 4024 to build multiple layers of GCN and self-attention layers to increase the representation capability of the model.

[0095] The final node representation can be the output H of the last GCN and self-attention layer (L) , where L is the index of the last layer.

[0096] The above scheme combines the node representation learning model of GCN and self-attention mechanism to allow nodes to propagate information in the graph and dynamically adjust according to their neighbors in the graph.

[0097] Step 403: concatenate the local features and the global features to obtain the final node representation vector;

[0098] Through the above technical solution, data scattered in different units and systems may lead to the formation of information islands, hinder information sharing and communication between different departments, and affect the consistency and efficiency of cross-departmental collaboration and decision-making. The present invention uses the perspective of graph neural networks to solve database matching problems and solve problems such as information islands and communication barriers.

[0099] Step 5, fusion matching: Based on the representation vector of the node, the Jaccard similarity between each pair of nodes is calculated, and the field matching is achieved through node matching using the Jaccard similarity to complete the fusion matching of the database;

[0100] In step 5, the Jaccard similarity between each pair of nodes is calculated based on the node representation vector, and the field matching is achieved by node matching using the Jaccard similarity, which specifically includes:

[0101] The node-based representation vector calculates the Jaccard similarity between each pair of nodes, and uses the Jaccard similarity to achieve field matching through node matching, including: calculating each node in different relationship graphs, calculating the set of neighbor nodes of the node, and obtaining the set of nodes adjacent to the node; using the Jaccard similarity formula to calculate the similarity of nodes in any two different relationship graphs, and using the node with the highest similarity as the matching node until a matching node is selected for the node in each relationship graph.

[0102] The following further explains using two graphs G1 and G2 as examples.

[0103] Step 501: For each node v in graph G1 1i , calculate the set N of its neighbor nodes 1i That is, with node v 1i A collection of connected nodes.

[0104] Similarly, for each node v in graph G2 2i , calculate the set N of its neighbor nodes 2i . Use the Jaccard similarity formula to calculate the similarity between two nodes. For example, for node v 1i and v 2i :

[0105]

[0106] Among them N1 i ∩N2 i Represents node v 1i and v 2i Number of common neighbor nodes, N1 i ∪N2 i Represents node v 1i and v 2i The union of the number of neighbor nodes.

[0107] Step 502: For each node v in the node set V1 c1j , select the node v with the highest Jaccard similarity in V2 c2j As a matching node:

[0108]

[0109] This means that the selection is related to v 1i The v2 node with the highest Jaccard similarity is taken as a match.

[0110] Step 503: Repeat steps 501 to 502 to select a matching node for each node in graph G1.

[0111] Step 6: Establish a unified data view based on the fused and matched database;

[0112] Establishing a unified view means identifying similar or identical nodes in different databases. For successfully matched node pairs, their attributes and related information are accurately matched and compared, and then a new node is established based on the attributes and related information of these successfully matched nodes; for nodes that are not successfully matched, their original node attributes and related information are retained. These successfully matched new nodes and unmatched old nodes form a new relationship graph, and a new database is established based on the new relationship graph. Then the staff can use this new database to extract the data view required for their work.

[0113] Through the above scheme, the database fusion matching method based on multi-dimensional indicator association analysis improves the field matching accuracy, establishes a unified data view, and helps developers query different databases according to needs in different applications, solving the fragmentation and dispersion of data, increasing the complexity of data access and management, and reducing the availability and accessibility of data; data inconsistency and inaccuracy bring uncertainty and risks to corporate decision-making and business operations; redundant storage and repeated collection of data increase data management and storage costs, and waste storage resources.

[0114] The domain name dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1, wherein the training set is used for model training, the validation set is used for hyperparameter determination, and the test set is used to verify model performance. Table 2 shows the experimental results comparison between the method of the present invention and the existing method.

[0115] Table 2 Performance comparison of fusion matching methods

[0116] VF2 Ullmann Method of the present invention Training loss 0.097 0.089 0.086 Accuracy 0.897 0.912 0.934 Recall 0.991 0.993 0.998

[0117] According to the results in Table 2, the database fusion matching method based on multidimensional indicator association analysis proposed in the present invention performs better than other methods.

[0118] Embodiment 2

[0119] like Figure 2 As shown, this embodiment provides a database fusion matching system based on multi-dimensional indicator association analysis, including:

[0120] A database acquisition module, which is used to acquire different types of power databases and database tables;

[0121] A metadata information extraction module, which is used to extract metadata information from different types of power databases;

[0122] A relationship graph generation module is used to convert the field information of the database into graph node information according to the metadata information in the database table and the extracted different types of power databases, respectively construct table entities, attribute entities and table-attribute relationships, and generate a relationship graph;

[0123] The node representation module is used to extract local features and global features in the relationship graph based on the node attributes and the connections between nodes in the relationship graph through the graph convolution network and the self-attention mechanism, and then concatenate the local features and the global features to obtain the node representation vector;

[0124] The fusion matching module is used to calculate the Jaccard similarity between each pair of nodes based on the node representation vector, and adopts the Jaccard similarity to achieve field matching through node matching to complete the fusion matching of the database.

[0125] The system also includes a data preprocessing module, which is used to preprocess the data of each database after acquiring different types of power databases, including data cleaning, missing value completion, data definition and storage.

[0126] The system also includes a data view module, which is used to identify similar or identical nodes in different databases. For successfully matched node pairs, their attributes and related information are accurately matched and compared, and then a new node is established based on the attributes and related information of these successfully matched nodes; for unmatched nodes, their original node attributes and related information are retained, and these successfully matched new nodes and unmatched old nodes form a new relationship graph, a new database is established based on the new relationship graph, and the required data view is extracted based on the new database.

[0127] The above technical solution can solve problems such as information islands and communication barriers. At the same time, it can help developers to query different databases according to needs in different applications, reduce the learning cost of developers for multiple system databases, and improve the generalization of multiple storage systems.

[0128] Embodiment 3

[0129] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the database fusion matching method based on multi-dimensional indicator association analysis as described in the first embodiment are implemented.

[0130] Embodiment 4

[0131] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the database fusion matching method based on multidimensional indicator association analysis as described in Embodiment 1 are implemented.

[0132] Embodiment 5

[0133] This embodiment provides a program product, which is a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps in the database fusion matching method based on multidimensional indicator association analysis as described in Example 1 are implemented.

[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A database fusion matching method based on multi-dimensional index association analysis, characterized in that: The steps include: Get different types of power databases and database tables; Extract metadata information from different types of power databases; According to the database table and the metadata information extracted from different types of power databases, the field information of the database is converted into graph node information, and table entities, attribute entities and table-attribute relationships are constructed respectively to generate a relationship graph; Based on the node attributes and the connections between nodes in the relationship graph, the local features and global features in the relationship graph are extracted through the graph convolution network and the self-attention mechanism. After splicing the local features and the global features, the representation vector of the node is obtained; Based on the representation vector of the node, the Jaccard similarity between each pair of nodes is calculated. The Jaccard similarity is used to achieve field matching through node matching to complete the fusion matching of the database.

2. A database fusion matching method based on multidimensional indicator association analysis as claimed in claim 1, characterized in that: The different types of power databases include Oracle database, Mysql database, Mongodb database, HDFS database and Hive database; The metadata includes field information, structural information, and data sample information; the field information includes name, annotation, structure, and constraints; the structural information includes the table to which it belongs and other fields and table information associated with the graph; and the data sample information includes the distribution, type, common features, and patterns of data values.

3. The database fusion matching method based on multi-dimensional index association analysis according to claim 1, characterized in that: Based on the node attributes in the relationship graph and the connections between the nodes, the local features and global features in the relationship graph are extracted through the graph convolution network and the self-attention mechanism, including: Combine the node attributes and the connections between nodes in the relationship graph, and use the graph convolutional neural network to extract the local features of the nodes in the relationship graph; After the output of each GCN layer in the graph convolutional neural network, a self-attention mechanism is introduced to learn the weights between nodes to capture the relationship between nodes, so as to further optimize the node representation and obtain global features; wherein, the introduction of the self-attention mechanism and the learning of the weights between nodes to capture the relationship between nodes include: Calculate the self-attention weight, calculate the normalized attention weight, use the attention weight to aggregate neighbor information, update the node representation based on the vector after aggregating neighbor information, and obtain the global feature.

4. The database fusion matching method based on multi-dimensional index association analysis according to claim 1, characterized in that: In the relationship graph, local features include: node degree, neighbor information and clustering coefficient, and global features include: graph density, average shortest path length, network diameter and network connectivity.

5. The database fusion matching method based on multi-dimensional index association analysis according to claim 1, characterized in that: The node-based representation vector is used to calculate the Jaccard similarity between each pair of nodes, and the field matching is achieved by node matching using the Jaccard similarity, including: Calculate each node in different relationship graphs, calculate the set of neighbor nodes of the node, and obtain the set of nodes adjacent to the node; The Jaccard similarity formula is used to calculate the similarity of nodes in any two different relationship graphs, and the nodes with the highest similarity are used as matching nodes until matching nodes are selected for the nodes in each relationship graph.

6. The database fusion matching method based on multi-dimensional index association analysis according to claim 1, characterized in that: The method also includes: identifying similar or identical nodes in different databases, accurately matching and comparing the attributes and related information of successfully matched node pairs, and then establishing a new node based on the attributes and related information of these successfully matched nodes; for nodes that are not successfully matched, retaining their original node attributes and related information, these successfully matched new nodes and unmatched old nodes form a new relationship graph, establishing a new database based on the new relationship graph, and extracting the required data view based on the new database.

7. A database fusion matching system based on multi-dimensional index association analysis, characterized in that: include: A database acquisition module, which is used to acquire different types of power databases and database tables; A metadata information extraction module, which is used to extract metadata information from different types of power databases; A relationship graph generation module is used to convert the field information of the database into graph node information according to the metadata information in the database table and the extracted different types of power databases, respectively construct table entities, attribute entities and table-attribute relationships, and generate a relationship graph; The node representation module is used to extract local features and global features in the relationship graph based on the node attributes and the connections between nodes in the relationship graph through the graph convolution network and the self-attention mechanism, and then concatenate the local features and the global features to obtain the node representation vector; The fusion matching module is used to calculate the Jaccard similarity between each pair of nodes based on the node representation vector, and adopts the Jaccard similarity to achieve field matching through node matching to complete the fusion matching of the database.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a database fusion matching method based on multidimensional indicator association analysis as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the database fusion matching method based on multidimensional indicator association analysis as described in any one of claims 1 to 6 are implemented.

10. A program product, the program product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of a database fusion matching method based on multidimensional indicator association analysis as described in any one of claims 1 to 6 are implemented.

Images