A cross-source data joint query method based on a data middle platform

Abstract

This invention relates to a cross-source data joint query method based on a data middle platform, comprising the following steps: S1: Constructing a data middle platform access strategy model to automatically generate the most suitable access method; S2: Constructing a power data knowledge graph and using graph neural networks to analyze the dependencies, tracing paths, and lineage relationships between data; S3: Achieving seamless access through natural language understanding, using natural language query translation to generate corresponding SQL statements from user natural language queries; S4: Calculating similarity based on the SQL statements using embedding vectors to match user questions with data models, and generating a virtual data view based on historical query patterns and business needs; S5: Based on the generated virtual data view, identifying potentially sensitive information fields in the dataset through natural language models and regular expression pattern recognition, and automatically labeling and classifying them. This invention significantly lowers the technical threshold for data fusion and sharing, and enhances the intelligence of the power industry data middle platform system.

Description

Technical Field

[0001] This invention relates to the field of data management using data middleware, and more particularly to a cross-source data joint query method based on data middleware. Background Technology

[0002] With the advancement of enterprise digital transformation and business diversification, the number and types of internal and external data sources are rapidly increasing, encompassing various forms such as traditional relational databases, unstructured and semi-structured data storage, distributed big data platforms, cloud SaaS systems, and IoT devices. Data is not only distributed across different systems and departments, but also exhibits significant differences in interface protocols, data formats, access permissions, and real-time requirements. In traditional data integration and query scenarios, developers typically need to develop separate interface programs for each data source, periodically extracting and aggregating data through ETL processes. This is inefficient, results in multiple data storage copies, and fails to meet real-time business decision-making and innovation needs. Summary of the Invention

[0003] To address the aforementioned issues, the present invention aims to provide a cross-source data joint query method based on a data platform, which significantly reduces the technical threshold for data fusion and sharing, improves the automation, intelligence, and security compliance level of the power industry data platform system, and provides scalable technical support for the efficient utilization of cross-system data.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A cross-source data joint query method based on a data middle platform includes the following steps:

[0006] S1: Construct a data middle platform access strategy model to automatically generate the most suitable access method based on data volume, interface stability and access requirements, and achieve adaptive selection and configuration.

[0007] S2: Construct a knowledge graph of power data, use graph neural networks to analyze the dependencies, tracing paths and lineages between data, realize automated lineage tracing, and automatically identify the impact chain when data is updated to assist in data quality control;

[0008] S3: Enables seamless access through natural language understanding. It uses natural language query translation to automatically generate corresponding SQL statements based on field semantics, data structure, and context from the user's natural language query.

[0009] S4: Based on the SQL statement, use embedding vectors to calculate similarity, match user questions with data models, ensure that the query is closer to the user's intent, and generate a virtual data view based on historical query patterns and business needs;

[0010] S5: Based on the generated virtual data view, it uses natural language models and regular pattern recognition to identify sensitive information fields that may be contained in the dataset, and performs automatic annotation and classification.

[0011] Furthermore, a data platform access strategy model is constructed to automatically generate access methods based on data volume, interface stability, and access requirements, enabling adaptive selection and configuration, as detailed below:

[0012] First, descriptive information is extracted from existing and potential data sources to form a multi-dimensional access feature set; the multi-dimensional access feature set is automatically extracted from historical access logs, monitoring data and external sensing systems, and input in a structured form;

[0013] Based on the above feature data, an access strategy model is constructed. The goal of the model is to evaluate the access cost and performance benefits of different data sources and automatically select the optimal access mode.

[0014] Ultimately, based on the prediction results, an access configuration file is automatically generated and dynamically deployed to the data platform access layer to achieve adaptive access.

[0015] Furthermore, the access strategy model is built based on machine learning and reinforcement learning, as detailed below:

[0016] The data source access strategy selection problem is defined as a multi-objective optimization Markov decision process, where the set of data sources is D = {d1, d2, ..., d...}. i ,...,d n The set of access strategies is A={a1,a2,a3,a4}, which correspond to four modes: real-time direct connection, incremental synchronization, asynchronous caching, and API aggregation, respectively.

[0017] For any data source d i Its eigenvectors are represented as:

[0018] x i =[D i ,S i A i E i ];

[0019] Each subvector has a dimension of 4, D i ,S i A i E i These represent data volume, stability, access requirements, and environmental characteristics, respectively.

[0020] The access strategy model adopts a three-layer decision structure, including a pre-trained classification layer, a reinforcement learning optimization layer, and a multi-objective decision fusion layer. The overall decision function is defined as follows:

[0021] ;

[0022] Among them, P base The basic classifier outputs the probability, Q. RL To reinforce the learning of the Q-value function, U MO Let α+β+γ=1 be the weight coefficients for the multi-objective optimization utility function.

[0023] Furthermore, the pre-trained classification layer uses the XGBoost model, as detailed below:

[0024] For the original feature matrix X={x1,x2,...,x} i ,...,x n Standardization processing is performed:

[0025] ;

[0026] Among them, X norm The matrix is ​​the standardized feature matrix; u is the mean vector, and σ is the standard deviation vector.

[0027] Principal component analysis was used for dimensionality reduction, retaining principal components whose cumulative variance contribution rate reached a preset value.

[0028] X pca =X norm *W pca ;

[0029] Among them, W pca Principal component transformation matrix;

[0030] The objective function of the XGBoost model is:

[0031] ;

[0032] in, Let be the loss function, the loss function for the i-th sample, and y be the measure of the true label. i Compared with model predictions The difference between them; G is the total number of all base learners in the XGBoost model; For the regularization term of the g-th basic learner;

[0033] For a four-class classification problem, the softmax function is used to calculate the selection probability of each strategy:

[0034] ;

[0035] in, Based on the classification model, the j-th access strategy a j The predicted probability;

[0036] f j (X pca For the classification model, the input X pca The output score for the j-th class; k is the policy category index; The normalized sum of the score indices for all access strategies;

[0037] The classification confidence score is defined as the ratio of the highest probability to the second highest probability.

[0038] ;

[0039] When Conf base (X pca )>τ conf , τ conf The confidence threshold is determined by directly using the results of the base classifier; where Confbase(Xpca) is the base classifier for sample X. pca The confidence level of the classification decision; It is the maximum value among the four access strategy probabilities, that is, the probability of the most likely category as determined by the model; This is the second most likely.

[0040] Furthermore, the reinforcement learning optimization layer and the multi-objective decision fusion layer are detailed below:

[0041] The reinforcement learning optimization layer models access policy selection as an MDP quadruple (S, A, P, R): State space S: s t =[x t ,h t ,l t ], x t ,h t ,l t These are the current characteristics, historical success rate, and system load, respectively; the action space A includes four access strategies {a1, a2, a3, a4}; the transition probability P adopts the state transition probability P(s). t+1 |s t ,a t ); Reward function R: R(s) t ,a t (This is an instant reward.)

[0042] Design a comprehensive reward function to balance performance gains and cost:

[0043] ;

[0044] ;

[0045] ;

[0046] Where w1, w2, w3, and w4 are the weights of each performance indicator; c1, c2, and c3 are the weights of the cost item; R(s) t ,a t (a) is the action to be taken at time t. t , in state s t The immediate reward obtained at that time; α1 and α2 are positive weighting coefficients, balancing performance gains and cost consumption; Perf(s t ,a t ) is in state s t Next, execute action a t Performance gains over time; Cost(s) t ,a t ) is in state s t Next, execute action a t Cost function at time T; max The maximum allowable response time limit; T response R represents the actual response latency; R represents the stability score of the system or connection; R represents the data accuracy score; R represents the resource utilization efficiency score; Rcompute represents the computational resource consumption; Rnetwork represents the network bandwidth consumption; and Rmaintain represents the human and operational resource consumption required for system maintenance.

[0047] The Q-value function is approximated using a deep neural network:

[0048] ;

[0049] Where σ is the ReLU activation function, θ={W1,W2,W3,b1,b2,b3} are network parameters; W1,W2,W3 are the weight matrices of the first, second, and third layers of the model, respectively; b1,b2,b3 are the bias vectors of each layer; and s is the vector representation of the state.

[0050] ;

[0051] ;

[0052] ;

[0053] in, is the optimal state-action value function; γ is the discount factor, 0 < γ < 1, which controls the impact of future rewards on the current value; s′ is the next state after executing a; a′ is all possible actions in the next state; δ is the expected average value; t For timing difference error; R t The immediate reward at time t; For target network parameters; This is the loss function used to train the Q-network;

[0054] Exploration is performed using an ε-greedy strategy:

[0055] ;

[0056] in, Let be the probability of choosing action a given state s; For exploration rate; The total number of actions in the space; The action with the largest Q value;

[0057] The multi-objective decision fusion layer ultimately selects a strategy based on maximizing expected utility.

[0058] ;

[0059] in, This is the sigmoid function, used to map Q values ​​to the [0,1] interval; The final selected optimal access strategy; , These are the weighting coefficients; The probability predicted by the XGboost model for access strategy a; To reinforce the learning model Q network for X pca The value estimation of action a.

[0060] Furthermore, a knowledge graph of power data is constructed, and graph neural networks are used to analyze the dependencies, tracing paths, and lineage relationships between data to achieve automated lineage tracing. When data is updated, the influencing chain is automatically identified to assist in data quality control, as detailed below:

[0061] ETL tools are used to automate the extraction of structured data, while rule mining and natural language processing techniques are used to identify entities and relationships in semi-structured and unstructured data. Automated scripts are used to map tables, fields, and process nodes in the data source to graph nodes, and various relationships in the ontology model are instantiated as edges. Graph databases are selected for efficient and scalable storage, and each node and edge is accompanied by detailed attributes.

[0062] Each node in the knowledge graph is encoded as a feature vector, and the edges between nodes represent dependencies. Adjacency moments are used as the input to the GNN.

[0063] Based on the node and link embedding information output by GNN, reverse tracing is performed on any data node to construct a lineage path. Recursive lookup and multi-hop diffusion are applied to the outward dependencies of any data table, field, and interface to automatically generate its downstream influence chain.

[0064] By analyzing triggers and change logs, we can monitor data update operations of data tables, fields, and interfaces in real time. When a data node is updated, we can use the node embedding and edge weight information of GNN to quickly retrieve all downstream nodes that are directly and indirectly affected, and evaluate the probability of each node being affected, the intensity of the impact, and the path of the impact.

[0065] Furthermore, seamless access is achieved through natural language understanding. Natural language query translation is used to automatically generate corresponding SQL statements from the user's natural language query based on field semantics, data structure, and context, as detailed below:

[0066] By introducing natural language understanding and semantic recognition, the system can automatically identify query intent (such as statistics, filtering, comparison) and key entities by semantic modeling of query statements. It can also parse the query based on semantic context and business domain knowledge, so that natural language queries can be converted into executable data access operations, thereby eliminating language and structural barriers between users and data systems.

[0067] In the implementation process, a pre-trained language model is first used to perform semantic embedding and intent recognition on user queries, automatically extracting query objects, filtering conditions, and aggregation logic. Then, the identified natural language entities are semantically mapped to the fields of the underlying data model: through field vector similarity matching, knowledge graph relationship reasoning, and structure verification, the required tables and field relationships are accurately located. After semantic parsing is completed, the system calls the Seq2SQL model or template-driven generation engine to automatically convert the natural language statements into SQL query statements.

[0068] Furthermore, based on the SQL statement, similarity is calculated using embedding vectors to match the user's question with the data model, ensuring that the query is closer to the user's intent. And based on historical query patterns and business needs, a virtual data view is generated, as follows:

[0069] Using deep semantic coding models (such as Sentence-BERT, SimCSE, or Transformer-based embedding models), SQL statements and data model field, table, view names, and annotation information are uniformly vectorized; each data entity (field, table, or object) forms a high-dimensional vector in the embedding space, and the semantic distance between vectors (measured by cosine similarity or Euclidean distance) is used to measure semantic relevance.

[0070] By capturing and analyzing users' historical query logs, a query pattern feature library is constructed, including common field combinations, join structures, filtering conditions, and time range characteristics. Subsequently, cluster analysis or a query pattern recognition network based on an attention mechanism is used to learn the typical query behaviors of different business roles in different scenarios. When a new query request arrives, semantic matching is performed first, and then the field priority and association logic are adjusted based on the query history similarity score. After the matching is completed, a logical virtual data view is generated.

[0071] Furthermore, the virtual data view is specifically designed as follows: Based on semantic similarity and historical query behavior, it automatically selects the fields and data tables most relevant to user needs. Especially in the case of multi-source heterogeneity, it can accurately select corresponding fields across systems and databases, and automatically identify and generate necessary table joins, field transformations, and alias mappings. On this basis, based on the filtering conditions, aggregation logic, and business constraints in the user query, it automatically adds filters and aggregation structures to the virtual view, and defines the field order, grouping, and hierarchy of the output data according to business requirements.

[0072] Furthermore, based on the generated virtual data view, natural language models and regular expression pattern recognition are used to identify potentially sensitive information fields in the dataset, and then automatically label and classify them, as follows:

[0073] After the system generates a virtual data view, the data platform further identifies and labels the sensitive information fields it contains. First, it establishes an identification knowledge base containing sensitive data features, structural rules, and domain lexicons, and uses regular expression matching to identify fields with typical pattern features. At the same time, for non-regular and semantically implicit field content, a pre-trained natural language model is used to extract semantic features to understand the meaning of the field in the context. The semantic recognition results are then fused with the regular expression detection results for scoring to determine the sensitivity level of the field.

[0074] After identifying sensitive fields, the data is automatically labeled and classified according to its sensitivity, dividing the fields into three categories: highly sensitive, generally sensitive, and low-risk. The classification results are written into the metadata tags of the virtual data view.

[0075] The present invention has the following beneficial effects:

[0076] 1. This invention constructs a data middle platform access strategy model, which intelligently selects and automatically configures access methods based on multi-dimensional parameters such as data volume, interface stability, and access requirements. This enables flexible access and real-time scheduling of data sources. Furthermore, by utilizing power data knowledge graphs and graph neural networks, it automatically perceives the dependencies and tracing relationships between different data sources and business objects, enabling intelligent analysis of lineage tracing and impact chains. When underlying data changes, the system can immediately identify the affected data range and automatically initiate relevant quality control processes, effectively ensuring the accuracy, continuity, and high availability of joint data queries and operations.

[0077] 2. This invention introduces natural language query translation, allowing users to express complex data needs simply through natural language. The system automatically parses user intent based on field semantics, data structure, and context, and intelligently generates corresponding SQL queries. Through embedding vectors and semantic similarity calculation, the system can not only more accurately match user query needs with the underlying data model, but also flexibly generate virtual data views by combining historical query habits and business context.

[0078] 3. In the data view returned by the joint query, the present invention integrates natural language model and regular expression pattern recognition to actively identify sensitive fields such as ID card number, email address, and transaction account from the dataset, and automatically completes the labeling and classification, effectively preventing the risk of missing labeling or misuse of sensitive information. Attached Figure Description

[0079] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0080] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0081] refer to Figure 1 In this embodiment, a cross-source data joint query method based on a data middle platform is provided, including the following steps:

[0082] S1: Construct a data middle platform access strategy model to automatically generate the most suitable access method based on data volume, interface stability and access requirements, and achieve adaptive selection and configuration.

[0083] S2: Construct a knowledge graph of power data, use graph neural networks to analyze the dependencies, tracing paths and lineages between data, realize automated lineage tracing, and automatically identify the impact chain when data is updated to assist in data quality control;

[0084] S3: Enables seamless access through natural language understanding. It uses natural language query translation to automatically generate corresponding SQL statements based on field semantics, data structure, and context from the user's natural language query.

[0085] S4: Based on the SQL statement, use embedding vectors to calculate similarity, match user questions with data models, ensure that the query is closer to the user's intent, and generate a virtual data view based on historical query patterns and business needs;

[0086] S5: Based on the generated virtual data view, it uses natural language models and regular expression pattern recognition to identify sensitive information fields (such as ID card numbers, email addresses, transaction numbers, etc.) that may be contained in the dataset, and performs automatic labeling and classification.

[0087] In this embodiment, a data platform access strategy model is constructed, which automatically generates access methods based on data volume, interface stability, and access requirements, achieving adaptive selection and configuration, as detailed below:

[0088] First, descriptive information is extracted from existing and potential data sources to form a multi-dimensional access feature set. This feature set includes, but is not limited to: data volume features: recording the data scale, data update frequency, and historical growth trends of each source; interface stability features: including indicators such as response latency, error rate, connection stability, and request success rate; access requirement features: covering factors such as access frequency, real-time requirements, query complexity, and downstream dependencies; and system environment features: network bandwidth, node location, authentication methods, and transmission protocols. The multi-dimensional access feature set is automatically extracted from historical access logs, monitoring data, and external sensor systems, and input in a structured form.

[0089] Based on the above feature data, an access strategy model is constructed. The goal of the model is to evaluate the access cost and performance benefits of different data sources and automatically select the optimal access mode.

[0090] Ultimately, based on the prediction results, an access configuration file is automatically generated and dynamically deployed to the data platform access layer to achieve adaptive access.

[0091] In this embodiment, the access strategy model is built based on machine learning and reinforcement learning, as detailed below:

[0092] The data source access strategy selection problem is defined as a multi-objective optimization Markov decision process, where the set of data sources is D = {d1, d2, ..., d...}. i ,...,d n The set of access strategies is A={a1,a2,a3,a4}, which correspond to four modes: real-time direct connection, incremental synchronization, asynchronous caching, and API aggregation, respectively.

[0093] For any data source d i Its eigenvectors are represented as:

[0094] x i =[Di ,S i A i E i ];

[0095] Each subvector has a dimension of 4, D i ,S i A i E i These represent data volume, stability, access requirements, and environmental characteristics, respectively.

[0096] The access strategy model adopts a three-layer decision structure, including a pre-trained classification layer, a reinforcement learning optimization layer, and a multi-objective decision fusion layer. The overall decision function is defined as follows:

[0097] ;

[0098] Among them, P base The basic classifier outputs the probability, Q. RL To reinforce the learning of the Q-value function, U MO Let α+β+γ=1 be the weight coefficients for the multi-objective optimization utility function.

[0099] In this embodiment, the pre-trained classification layer uses the XGBoost model, as detailed below:

[0100] For the original feature matrix X={x1,x2,...,x} i ,...,x n Standardization processing is performed:

[0101] ;

[0102] Among them, X norm The matrix is ​​the standardized feature matrix; u is the mean vector, and σ is the standard deviation vector.

[0103] Principal component analysis was used for dimensionality reduction, retaining principal components whose cumulative variance contribution rate reached a preset value.

[0104] X pca =X norm *W pca ;

[0105] Among them, W pca Principal component transformation matrix;

[0106] The objective function of the XGBoost model is:

[0107] ;

[0108] in, Let be the loss function, the loss function for the i-th sample, and y be the measure of the true label.i Compared with model predictions The difference between them; G is the total number of all base learners (such as regression trees) in the XGBoost model; For the g-th basic learner (such as the k-th tree);

[0109] For a four-class classification problem, the softmax function is used to calculate the selection probability of each strategy:

[0110] ;

[0111] in, Based on the classification model, the j-th access strategy a j The predicted probability;

[0112] f j (X pca For the classification model, the input X pca The output score for the j-th class; k is the policy category index; The normalized sum of the score indices for all access strategies;

[0113] The classification confidence score is defined as the ratio of the highest probability to the second highest probability.

[0114] ;

[0115] When Conf base (X pca )>τ conf , τ conf The confidence threshold is determined by directly using the results of the base classifier; where Confbase(Xpca) is the base classifier for sample X. pca The confidence level of the classification decision; It is the maximum value among the four access strategy probabilities, that is, the probability of the most likely category as determined by the model; This is the second most likely.

[0116] In this embodiment, the reinforcement learning optimization layer and the multi-objective decision fusion layer are specifically as follows:

[0117] The reinforcement learning optimization layer models access policy selection as an MDP quadruple (S, A, P, R): State space S: s t =[x t ,h t ,l t ], x t ,h t ,l t These are the current characteristics, historical success rate, and system load, respectively; the action space A includes four access strategies {a1, a2, a3, a4}; the transition probability P adopts the state transition probability P(s).t+1 |s t ,a t ); Reward function R: R(s) t ,a t (This is an instant reward.)

[0118] Design a comprehensive reward function to balance performance gains and cost:

[0119] ;

[0120] ;

[0121] ;

[0122] Where w1, w2, w3, and w4 are the weights of each performance indicator; c1, c2, and c3 are the weights of the cost item; R(s) t ,a t (a) is the action to be taken at time t. t , in state s t The immediate reward obtained at that time; α1 and α2 are positive weighting coefficients, balancing performance gains and cost consumption; Perf(s t ,a t ) is in state s t Next, execute action a t Performance gains over time; Cost(s) t ,a t ) is in state s t Next, execute action a t Cost function at time T; max The maximum allowable response time limit; T response R represents the actual response latency; R represents the stability score of the system or connection; R represents the data accuracy score; R represents the resource utilization efficiency score; Rcompute represents the computing resource consumption (such as normalized values ​​of CPU, memory, and other metrics); Rnetwork represents the network bandwidth consumption; and Rmaintain represents the manpower and maintenance resource consumption required for system maintenance.

[0123] The Q-value function is approximated using a deep neural network:

[0124] ;

[0125] Where σ is the ReLU activation function, θ={W1,W2,W3,b1,b2,b3} are network parameters; W1,W2,W3 are the weight matrices of the first, second, and third layers of the model, respectively; b1,b2,b3 are the bias vectors of each layer; and s is the vector representation of the state.

[0126] ;

[0127] ;

[0128] ;

[0129] in, is the optimal state-action value function; γ is the discount factor, 0 < γ < 1, which controls the impact of future rewards on the current value; s′ is the next state after executing a; a′ is all possible actions in the next state; δ is the expected average value; t For timing difference error (TD error); R t The immediate reward at time t; For target network parameters; This is the loss function used to train the Q-network;

[0130] Exploration is performed using an ε-greedy strategy:

[0131] ;

[0132] in, Let be the probability of choosing action a given state s; For exploration rate; The total number of actions in the space; The action with the largest Q value;

[0133] The multi-objective decision fusion layer ultimately selects a strategy based on maximizing expected utility.

[0134] ;

[0135] in, This is the sigmoid function, used to map Q values ​​to the [0,1] interval; The final selected optimal access strategy; , These are the weighting coefficients; The probability predicted by the XGboost model for access strategy a; To reinforce the learning model Q network for X pca The value estimation of action a.

[0136] In this embodiment, a power data knowledge graph is constructed, and graph neural networks are used to analyze the dependencies, tracing paths, and lineage relationships between data to achieve automated lineage tracing. When data is updated, the influencing chain is automatically identified to assist in data quality control, as detailed below:

[0137] For structured data, ETL tools are used for automated extraction, while for semi-structured and unstructured data, rule mining and natural language processing techniques are used for entity and relation identification. Automated scripts are used to map tables, fields, and process nodes in the data source to graph nodes, and various relations in the ontology model are instantiated as edges. Graph databases (such as Neo4j or TigerGraph) are selected for efficient and scalable storage, and each node and edge is accompanied by detailed attributes (such as timestamps, quality levels, data update history, etc.).

[0138] Each node in the knowledge graph (such as data tables, fields, interfaces, business objects, etc.) is encoded into a feature vector (including type encoding, number of dependencies, activity, historical anomalies, etc.), and the edges between nodes represent dependencies (such as computation chains, data retrieval chains, and call chains). Adjacency moments are used to express the GNN input.

[0139] Based on the node and link embedding information output by GNN, reverse tracing is performed on any data node (i.e., automatically finding its data source and all intermediate transformation nodes), and a lineage path is constructed to apply recursive search and multi-hop diffusion to the outward dependencies of any data table, field, and interface, automatically generating its downstream influence chain.

[0140] By analyzing triggers and change logs, we can monitor data update operations of data tables, fields, and interfaces in real time. When a data node is updated, we can use the node embedding and edge weight information of GNN to quickly retrieve all downstream nodes that are directly and indirectly affected, and evaluate the probability of each node being affected, the intensity of the impact, and the path of the impact.

[0141] In this embodiment, seamless access is achieved through natural language understanding. Natural language query translation is used to automatically generate corresponding SQL statements from the user's natural language query based on field semantics, data structure, and context, as detailed below:

[0142] By introducing Natural Language Understanding (NLU) and Semantic Recognition, the system can automatically identify query intent (such as statistics, filtering, and comparison) and key entities (such as device type, geographical range, and time conditions) by semantic modeling of query statements. It can also parse the query based on semantic context and business domain knowledge, so that natural language queries can be converted into executable data access operations, thereby eliminating language and structural barriers between users and data systems.

[0143] In the implementation process, pre-trained language models (such as BERT, GPT, or ERNIE) are first used to perform semantic embedding and intent recognition on user queries, automatically extracting query objects, filtering conditions, and aggregation logic. Then, the identified natural language entities are semantically mapped to the fields of the underlying data model: through field vector similarity matching, knowledge graph relationship reasoning, and structure verification, the required tables and field relationships are accurately located. After semantic parsing, the system calls the Seq2SQL model or template-driven generation engine to automatically convert the natural language statement into an SQL query statement. For example, if the input is "Query the average load rate of substations in East China last month," the system automatically generates an SQL statement with time, region, indicator, and aggregation logic, and verifies and optimizes the syntax and executability to ensure accurate and logically complete query results.

[0144] In this embodiment, based on the SQL statement, similarity is calculated using embedding vectors to match the user's question with the data model, ensuring that the query is closer to the user's intent. Furthermore, based on historical query patterns and business requirements, a virtual data view is generated, as detailed below:

[0145] Using deep semantic coding models (such as Sentence-BERT, SimCSE, or Transformer-based embedding models), SQL statements and data model field, table, view names, and annotation information are uniformly vectorized. Each data entity (field, table, or object) forms a high-dimensional vector in the embedding space, and the semantic distance between vectors (measured by cosine similarity or Euclidean distance) is used to measure semantic relevance. For example, when a user query contains "load statistics", the system can automatically identify its semantic proximity to multiple physical fields (such as load_value, active_power, station_load) and generate a candidate mapping list ranked by similarity.

[0146] By capturing and analyzing users' historical query logs, a query pattern feature library is constructed, including common field combinations, join structures, filtering conditions, and time range characteristics. Subsequently, clustering analysis or an attention-based query pattern recognition model is used to learn typical query behaviors of different business roles in different scenarios. When a new query request arrives, semantic matching is performed first, and then the field priority and association logic are adjusted based on the query history similarity score.

[0147] For example, for dispatchers, the system tends to recommend fields related to load_forecast and grid_balance; for equipment maintenance personnel, it prioritizes the equipment_status and fault_type field sets. This step ensures that the data access results are not only semantically correct but also have higher business relevance and personalized intelligence; after matching is completed, a logical-level virtual data view is generated.

[0148] In this embodiment, the virtual data view is defined as follows: Based on semantic similarity and historical query behavior, the fields and data tables most relevant to the user's needs are automatically selected. Especially in the case of multi-source heterogeneity, it can accurately select corresponding fields across systems and databases, and automatically identify and generate necessary table joins (such as JOIN conditions), field transformations, and alias mappings. On this basis, based on the filtering conditions, aggregation logic (such as grouping, statistics, and sorting) and business constraints (such as time, region, and object type) in the user's query, filters and aggregation structures are automatically added to the virtual view, and the field order, grouping, and hierarchy of the output data are defined according to business requirements.

[0149] In this embodiment, based on the generated virtual data view, sensitive information fields that may be contained in the dataset are identified through natural language models and regular expression pattern recognition, and then automatically labeled and classified, as follows:

[0150] After the system generates a virtual data view, the data platform further identifies and labels the sensitive information fields it contains to ensure security and compliance during data sharing, analysis, and use. To this end, the solution comprehensively utilizes an intelligent detection mechanism combining Natural Language Processing (NLP) and regular pattern recognition to perform multi-level analysis of field names, field content, and metadata information in the virtual data view. The system can identify typical sensitive information such as ID card numbers, mobile phone numbers, email addresses, bank card numbers, transaction serial numbers, and geographical locations, and complete automatic detection and accurate classification without relying on manual review, thereby achieving secure identification and dynamic protection of cross-source and cross-system data.

[0151] First, a recognition knowledge base is established, including sensitive data features, structural rules, and domain lexicons. Regular expression rules are used to identify fields with typical pattern features, such as length, character composition, or naming characteristics. Simultaneously, for non-regular and semantically implicit field content, pre-trained natural language models (such as BERT, ERNIE, or RoBERTa) are used to extract semantic features to understand the meaning of the field in context. The semantic recognition results are then fused with the regular expression detection results for scoring to determine the sensitivity level of the field. Through this dual analysis approach of "rules + semantics," the platform can not only quickly discover explicit sensitive data but also identify fields with ambiguous names, polysemous content, or implicit sensitive attributes, ensuring the accuracy and coverage of the recognition results.

[0152] After identifying sensitive fields, the system automatically labels and categorizes them based on their sensitivity, classifying them into three categories: highly sensitive, generally sensitive, and low-risk. The classification results are written into the metadata tags of the virtual data view, providing a basis for subsequent data anonymization, encryption, access control, and auditing. The system can automatically execute anonymization rules (such as masking, obfuscation, or hash encryption) according to the classification strategy, ensuring that sensitive information is protected during display, export, or analysis. Simultaneously, the system continuously optimizes the identification model through self-learning, constantly expanding the sensitive field feature library based on user feedback and new samples, improving its intelligent and adaptive capabilities. Thus, the data platform, while enabling natural querying and intelligent analysis, also ensures data security and compliant controllability, constructing a comprehensive sensitive data protection system.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A cross-source data joint query method based on a data middle platform, characterized in that, Includes the following steps: S1: Construct a data middle platform access strategy model to automatically generate the most suitable access method based on data volume, interface stability and access requirements, and achieve adaptive selection and configuration. S2: Construct a knowledge graph of power data, use graph neural networks to analyze the dependencies, tracing paths and lineages between data, realize automated lineage tracing, and automatically identify the impact chain when data is updated to assist in data quality control; S3: Enables seamless access through natural language understanding. It uses natural language query translation to automatically generate corresponding SQL statements based on field semantics, data structure, and context from the user's natural language query. S4: Based on the SQL statement, use embedding vectors to calculate similarity, match user questions with data models, ensure that the query is closer to the user's intent, and generate a virtual data view based on historical query patterns and business needs; S5: Based on the generated virtual data view, it uses natural language models and regular pattern recognition to identify sensitive information fields that may be contained in the dataset, and performs automatic annotation and classification. The access strategy model is built based on machine learning and reinforcement learning, as detailed below: The data source access strategy selection problem is defined as a multi-objective optimization Markov decision process, where the set of data sources is D = {d1, d2, ..., d...}. i ,...,d n The set of access strategies is A={a1,a2,a3,a4}, which correspond to four modes: real-time direct connection, incremental synchronization, asynchronous caching, and API aggregation, respectively. For any data source d i , Its eigenvectors are represented as follows: x i =[D i ,S i ,A i ,E i ]; Each subvector has a dimension of 4, D i ,S i A i E i These represent data volume, stability, access requirements, and environmental characteristics, respectively. The access strategy model adopts a three-layer decision structure, including a pre-training classification layer, a reinforcement learning optimization layer, and a multi-objective decision fusion layer, with the overall decision function... Defined as: ； Among them, P base The basic classifier outputs the probability, Q. RL To reinforce the learning of the Q-value function, U MO For multi-objective optimization utility function, α+β+γ=1 are weight coefficients; The pre-trained classification layer uses the XGBoost model, as detailed below: For the original feature matrix X={x1,x2,...,x} i ,...,x n Standardization processing is performed: ; Among them, X norm The matrix is ​​the standardized feature matrix; u is the mean vector, and σ is the standard deviation vector. Principal component analysis was used for dimensionality reduction, retaining principal components whose cumulative variance contribution rate reached a preset value. X pca =X norm W pca ; Among them, W pca Principal component transformation matrix; The objective function of the XGBoost model is: ; in, Let be the loss function, the loss function for the i-th sample, and y be the measure of the true label. i Compared with model predictions The difference between them; G is the total number of all base learners in the XGBoost model; For the regularization term of the g-th basic learner; For a four-class classification problem, the softmax function is used to calculate the selection probability of each strategy: ； in, Based on the classification model, the j-th access strategy a j The predicted probability; f j (X pca ) For the classification model to handle input X pca The output score for the j-th class; k is the policy category index; The normalized sum of the score indices for all access strategies; The classification confidence score is defined as the ratio of the highest probability to the second highest probability. ； When Conf base (X pca )>τ conf , τ conf The confidence threshold is determined by directly using the results of the base classifier; where Confbase(Xpca) is the base classifier for sample X. pca The confidence level of the classification decision; It is the maximum value among the four access strategy probabilities, that is, the probability of the most likely category as determined by the model; This is the second most likely.

2. The cross-source data joint query method based on a data middle platform according to claim 1, characterized in that, The data platform access strategy model is constructed by automatically generating access methods based on data volume, interface stability, and access requirements, achieving adaptive selection and configuration, as detailed below: First, descriptive information is extracted from existing and potential data sources to form a multi-dimensional access feature set; the multi-dimensional access feature set is automatically extracted from historical access logs, monitoring data and external sensing systems, and input in a structured form; Based on the above feature data, an access strategy model is constructed. The goal of the model is to evaluate the access cost and performance benefits of different data sources and automatically select the optimal access mode. Ultimately, based on the prediction results, an access configuration file is automatically generated and dynamically deployed to the data platform access layer to achieve adaptive access.

3. The method for cross-source data joint query based on a data middle platform according to claim 1, characterized in that, The reinforcement learning optimization layer and the multi-objective decision fusion layer are specifically as follows: The reinforcement learning optimization layer models access policy selection as an MDP quadruple (S, A, P, R): State space S: s t =[x t ,h t ,l t ], x t ,h t ,l t These are the current characteristics, historical success rate, and system load, respectively; the action space A includes four access strategies {a1, a2, a3, a4}; the transition probability P adopts the state transition probability P(s). t+1 |s t ,a t ); Reward function R: R(s) t ,a t (This is an instant reward.) Design a comprehensive reward function to balance performance gains and cost: ； ； ； Where w1, w2, w3, and w4 are the weights of each performance indicator; c1, c2, and c3 are the weights of the cost item; R(s) t ,a t (a) is the action to be taken at time t. t , in state s t The immediate reward obtained at that time; α1 and α2 are positive weighting coefficients, balancing performance gains and cost consumption; Perf(s t ,a t ) is in state s t Next, execute action a t Performance gains over time; Cost(s) t ,a t ) is in state s t Next, execute action a t Cost function at time T; max The maximum allowable response time limit; T response R represents the actual response latency; R represents the stability score of the system or connection; R represents the data accuracy score; R represents the resource utilization efficiency score; Rcompute represents the computational resource consumption; Rnetwork represents the network bandwidth consumption; and Rmaintain represents the human and operational resource consumption required for system maintenance. The Q-value function is approximated using a deep neural network: ； in, σ It is the ReLU activation function. θ={W 1 ,W 2 ,W 3 ,b 1 ,b 2 ,b 3 } For network parameters; W 1 ,W 2 ,W 3 These are the weight matrices for the first, second, and third layers of the model, respectively. b 1 ,b 2 ,b 3 Bias vectors for each layer; s is the vector representation of the state; ； ； ； Among them, Q The optimal state-action value function; γ is the discount factor, 0 < γ < 1, which controls the impact of future rewards on the current value; s′ is the next state after executing a; a′ is all possible actions in the next state; E[·] is the expected average value; δ t For timing difference error; R t θ represents the immediate reward at time t; - For target network parameters; This is the loss function used to train the Q-network; Exploration is performed using an ε-greedy strategy: ； in, Let be the probability of choosing action a given state s; For exploration rate; The total number of actions in the space; The action with the largest Q value; The multi-objective decision fusion layer ultimately selects a strategy based on maximizing expected utility. ； Where σ(·) is the sigmoid function, used to map Q values ​​to the interval [0,1]; The final selected optimal access strategy; , These are the weighting coefficients; The probability predicted by the XGboost model for access strategy a; To reinforce the learning model Q network for X pca The value estimation of action a.

4. The cross-source data joint query method based on a data middle platform according to claim 1, characterized in that, The aforementioned construction of a power data knowledge graph utilizes graph neural networks to analyze dependencies, tracing paths, and lineage relationships between data, enabling automated lineage tracing. When data is updated, it automatically identifies the influencing chains, assisting in data quality control, as detailed below: ETL tools are used to automate the extraction of structured data, while rule mining and natural language processing techniques are used to identify entities and relationships in semi-structured and unstructured data. Automated scripts are used to map tables, fields, and process nodes in the data source to graph nodes, and various relationships in the ontology model are instantiated as edges. Graph databases are selected for efficient and scalable storage, and each node and edge is accompanied by detailed attributes. Each node in the knowledge graph is encoded as a feature vector, and the edges between nodes represent dependencies. Adjacency moments are used as the input to the GNN. Based on the node and link embedding information output by GNN, reverse tracing is performed on any data node to construct a lineage path. Recursive lookup and multi-hop diffusion are applied to the outward dependencies of any data table, field, and interface to automatically generate its downstream influence chain. By analyzing triggers and change logs, we can monitor data update operations of data tables, fields, and interfaces in real time. When a data node is updated, we can use the node embedding and edge weight information of GNN to quickly retrieve all downstream nodes that are directly and indirectly affected, and evaluate the probability of each node being affected, the intensity of the impact, and the path of the impact.

5. The cross-source data joint query method based on a data middle platform according to claim 1, characterized in that, The seamless access achieved through natural language understanding employs natural language query translation, automatically generating corresponding SQL statements from the user's natural language query based on field semantics, data structure, and context, as detailed below: By introducing natural language understanding and semantic recognition, the system can automatically identify query intent and key entities by semantic modeling of query statements, and parse them based on semantic context and business domain knowledge, so that natural language queries can be transformed into executable data access operations, thereby eliminating language and structural barriers between users and data systems. In the implementation process, a pre-trained language model is first used to perform semantic embedding and intent recognition on user queries, and to automatically extract query objects, filtering conditions and aggregation logic. Subsequently, the identified natural language entities are semantically mapped to the fields of the underlying data model: through field vector similarity matching, knowledge graph relationship reasoning and structure verification, the required table and field relationship are accurately located; after semantic parsing is completed, the system calls the Seq2SQL model or template-driven generation engine to automatically convert the natural language statement into an SQL query statement.

6. The cross-source data joint query method based on a data middle platform according to claim 1, characterized in that, The process involves calculating similarity based on SQL statements using embedding vectors, matching user questions with data models to ensure queries better reflect user intent, and generating virtual data views based on historical query patterns and business requirements, as detailed below: Using a deep semantic coding model, SQL statements and data model field, table, view names, and comment information are uniformly vectorized; each data entity forms a high-dimensional vector in the embedding space, and the semantic distance between vectors is used to measure semantic relevance. By capturing and analyzing users' historical query logs, a query pattern feature library is constructed, including common field combinations, join structures, filtering conditions, and time range characteristics. Subsequently, cluster analysis or a query pattern recognition network based on an attention mechanism is used to learn the typical query behaviors of different business roles in different scenarios. When a new query request arrives, semantic matching is performed first, and then the field priority and association logic are adjusted by combining the query history similarity score. After the matching is completed, a virtual data view at the logical level is generated.

7. The cross-source data joint query method based on a data platform according to claim 6, characterized in that, The virtual data view is specifically configured as follows: Based on semantic similarity and historical query behavior, it automatically selects the fields and data tables most relevant to the user's needs. In the case of multi-source heterogeneity, it can accurately select corresponding fields across systems and databases, and automatically identify and generate necessary table joins, field transformations, and alias mappings. On this basis, according to the filtering conditions, aggregation logic, and business constraints in the user's query, it automatically adds filters and aggregation structures to the virtual view, and defines the field order, grouping, and hierarchy of the output data according to business requirements.

8. The cross-source data joint query method based on a data middle platform according to claim 1, characterized in that, The generated virtual data view uses natural language models and regular expression pattern recognition to identify potentially sensitive information fields in the dataset, and then automatically labels and classifies them, as follows: After the system generates a virtual data view, the data platform further identifies and labels the sensitive information fields it contains. First, it establishes an identification knowledge base containing sensitive data features, structural rules, and domain lexicons, and uses regular expression matching to identify fields with typical pattern features. At the same time, for non-regular and semantically implicit field content, a pre-trained natural language model is used to extract semantic features to understand the meaning of the field in the context. The semantic recognition results are then fused with the regular expression detection results for scoring to determine the sensitivity level of the field. After identifying sensitive fields, the data is automatically labeled and classified according to its sensitivity, dividing the fields into three categories: highly sensitive, generally sensitive, and low-risk. The classification results are written into the metadata tags of the virtual data view.

Images