Intelligent power grid-oriented trusted streaming data processing system and method

Abstract

The invention discloses a trusted streaming data processing system and method for a smart power grid, and belongs to the technical field of cloud computing and edge computing. In order to solve the security and privacy protection problems in the streaming data processing process in the edge network environment, the technical scheme of combining task feature extraction, dynamic algorithm decision, link encryption and differential privacy protection is mainly adopted. According to the method, data transmission credibility and privacy security can be ensured, and the system resource utilization rate and the streaming data processing efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of cloud computing and edge computing, and particularly relates to a trusted streaming data processing system and method for an intelligent power grid. Background Art

[0002] With the continuous development and wide application of edge computing technology, streaming data processing systems are gradually transitioning from cloud data centers to the edge side for edge computing-intensive scenarios such as intelligent power grids. In the intelligent power grid scenario, massive edge devices such as intelligent electricity meters and sensors continuously generate streaming data such as electricity consumption measurement and operating status. It is necessary to rely on a streaming data processing system to complete data cleaning, analysis and calculation, and training and prediction of real-time models, etc., to support applications such as continuous monitoring and summarization of device status, fault analysis, electricity load prediction, and low-voltage early warning.

[0003] However, edge devices such as intelligent electricity meters and sensors usually do not have the conditions to access private networks such as Ethernet and WiFi, and usually use mobile communication networks such as 4G and 5G to access. There are security risks and privacy leakage risks when transmitting data back to the streaming data processing platform across the public Internet. Existing streaming data processing systems usually focus on the efficiency of data processing, etc., and rarely consider the trustworthiness of the data processing process and data privacy protection. On the other hand, the data encryption and privacy protection process requires occupying some computing resources. Ignoring the execution characteristics of streaming processing tasks, simply using strategies such as encryption may lead to resource contention or resource waste, affecting the availability and performance of the system.

[0004] In summary, there is an urgent need for a trusted streaming data processing method and system for an intelligent power grid to add link encryption and privacy protection features to the streaming data processing process to achieve the trustworthiness of the streaming data processing process and ensure the reliability and security of the streaming data processing process. By perceiving the characteristics of the current streaming data processing task to guide the selection of link encryption and privacy protection algorithms, the resource utilization rate is optimized. Summary of the Invention

[0005] The purpose of the present invention is to propose a trusted streaming data processing system and method for an intelligent power grid, which can solve the security and privacy protection problems in the streaming data processing process in the edge network environment.

[0006] The technical solutions adopted by the present invention to achieve the above purpose are as follows:

[0007] A trusted streaming data processing system for an intelligent power grid includes the following steps:

[0008] A master node and multiple worker nodes, the master node is connected to each worker node through a network;

[0009] The master node is deployed with a decision-making module;

[0010] The worker node is deployed with a task feature extraction module, a streaming data processing module, a link encryption module, and a privacy protection module;

[0011] The task feature extraction module is used to monitor the status of the streaming data processing task and extract task features based on a dynamically adjusted time window strategy and data stream characteristics;

[0012] The decision-making module is used to receive the task features, select applicable encryption algorithms and privacy protection algorithms from a preset algorithm library based on the calculated adaptation scores, and send the selected algorithms to each worker node;

[0013] The streaming data processing module is used to receive the streaming computing operators sent by the master node and perform real-time data processing;

[0014] The link encryption module is used to encrypt and decrypt data during data transmission according to the algorithm instructions sent by the decision-making module;

[0015] The privacy protection module is used to add random noise to sensitive data or processing results during the streaming data processing to achieve privacy protection.

[0016] Furthermore, the task feature extraction module is also used for:

[0017] Continuously monitor the execution status of the streaming data processing task on this worker node;

[0018] Extract task feature vectors based on real-time streaming data characteristics and task status;

[0019] Generate a digital signature for the feature vector and upload it to the master node through a secure channel at a predetermined period.

[0020] Furthermore, the task feature extraction module is also used for:

[0021] Adopt a dynamically adjusted time window, where the window size is inversely proportional to the data flow rate, and predict the flow rate change trend through Kalman filtering;

[0022] Within each window period, use the KS test to compare the distribution differences between the current window and the historical window for numerical data; construct a feature matrix;

[0023] Calculate the feature weights through the multi-head attention mechanism and output the dynamically weighted feature vector as the task feature.

[0024] Furthermore, the task features include data traffic, distribution difference, processing delay, computing resource utilization rate, and data sensitivity index.

[0025] Further, the decision-making module is also used for:

[0026] Receiving and verifying the feature vectors and signatures from each worker node;

[0027] Calculating the adaptation scores of candidate algorithms based on the received feature vectors, the real-time load information of each worker node, and the network latency status;

[0028] Screening out the link encryption algorithm and privacy protection algorithm with the highest adaptation score from the preset algorithm library, and determining the algorithm identification and parameter configuration;

[0029] Encapsulating the algorithm identification and parameters into a decision instruction, and sending it to the corresponding worker node, while confirming the instruction reception status.

[0030] Further, the streaming data processing module is also used for:

[0031] Receiving the streaming computing operator and execution topology sent by the master node;

[0032] Sharding and scheduling the input data stream according to the execution topology;

[0033] Before each operator execution, calling the link encryption module to encrypt or decrypt the data block to be processed;

[0034] After the operator processing is completed, calling the privacy protection module to add differential privacy noise to sensitive data or intermediate results;

[0035] Outputting the processed data according to the specified path and reporting the processing status in real time.

[0036] Further, the link encryption module is also used for:

[0037] Receiving the encryption algorithm identification and key parameters sent by the decision-making module;

[0038] Obtaining or generating symmetric or asymmetric keys required for encryption / decryption from the key management unit;

[0039] Performing encryption of each data packet sent to the network using the specified algorithm, and decrypting the encrypted data packet at the receiving end;

[0040] Performing integrity verification on the decrypted data and feeding back the verification result to the streaming data processing module.

[0041] Further, the privacy protection module is also used for:

[0042] Receiving the privacy protection algorithm identification and noise parameters sent by the decision-making module;

[0043] Applying Laplace or Gaussian noise to sensitive fields at each stage of streaming processing;

[0044] Verify whether the data after adding noise meets the preset privacy budget and differential privacy ε-δ standard;

[0045] Transmit the data after privacy protection to the downstream module and record the noise usage and privacy budget consumption.

[0046] A trusted streaming data processing method for smart grid, comprising the following steps:

[0047] 1) The master node receives the streaming data processing task submitted by the user and the requirements for link encryption and privacy protection, and decomposes the streaming data processing task into multiple streaming computing operators;

[0048] 2) The master node distributes the streaming computing operators and the requirements for link encryption and privacy protection to the worker nodes;

[0049] 3) The worker node monitors the execution status of the received streaming computing operators, and extracts task characteristics including data traffic, processing delay, computing resource utilization rate, and data sensitivity;

[0050] 4) The worker node sends the task characteristics to the master node;

[0051] 5) Based on the task characteristics and the requirements for link encryption and privacy protection, the master node selects a link encryption algorithm and a privacy protection algorithm from a preset algorithm library, and distributes the selected algorithm identifier and parameter configuration to the worker nodes;

[0052] 6) During the streaming data processing, the worker node calls the link encryption algorithm to encrypt and decrypt the data transmission data packet, and calls the privacy protection algorithm to add random noise to sensitive data or intermediate results, and returns the processed data to the user through the master node.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] (1) The present invention designs a task characteristic calculation method for streaming data of a dynamically changing power grid with variable flow rate, and designs an algorithm scoring and selection mechanism based on task characteristics. Different from the prior art that uses fixed encryption / protection algorithms or methods for static data and fixed task characteristics, the present invention provides an adaptive trusted data processing solution for the dynamic changes of data flow rate, data distribution, and cluster resources. Through the dynamic algorithm selection and parameter optimization of the decision module, the present invention can realize the automatic switching of link encryption algorithms and privacy protection algorithms, improve the resource utilization rate and processing efficiency of the system, and solve the data processing problem in complex environments.

[0055] (2) The present invention encrypts and decrypts streaming data during data transmission through a link encryption module, ensuring the credibility and security of data transmission in a cross-public Internet environment and preventing data from being maliciously eavesdropped or tampered with.

[0056] (3) The present invention adds random noise that complies with the differential privacy standard to sensitive data or intermediate results through a privacy protection module, preventing malicious parties from inferring user privacy information through traffic or result analysis.

[0057] (4) The present invention realizes the collaborative scheduling and security processing of the master node and multiple worker nodes in an edge network environment, supports real-time streaming data processing and dynamic algorithm distribution, and improves the reliability, scalability, and maintainability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a module diagram of a trusted streaming data processing system for smart grid according to the present invention;

[0059] Figure 2 is a flowchart of a trusted streaming data processing method for smart grid according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] In order to make the technical features, advantages, or technical effects in the above technical solutions of the present invention more obvious and understandable, the following will be described in detail through embodiments and in conjunction with the drawings.

[0061] An embodiment of the present invention discloses a trusted streaming data processing system for smart grid, and its module composition is as Figure 1 shown, mainly including a task feature extraction module, a decision-making module, a streaming data processing module, a link encryption module, and a privacy protection module. The deployment, main responsibilities, and interaction methods of each module are as follows:

[0062] 1. The system cluster consists of a master node and multiple worker nodes. The master node is mainly responsible for the scheduling of streaming data processing tasks and algorithm decision-making, and the worker nodes are mainly responsible for extracting the feature data of the current task and performing link encryption and privacy protection operations according to the decision of the master node;

[0063] 2. The task feature extraction module is deployed on the worker node and is responsible for real-time monitoring and extracting the key features of the streaming data processing tasks executed on this node;

[0064] 3. The decision-making module is deployed on the master node and is responsible for collecting the task features extracted on each worker node and driving the selection of link encryption and privacy protection algorithms based on these features;

[0065] 4. The streaming data processing module is deployed on the worker node and is responsible for executing specific streaming data processing tasks, including real-time processing and calculation of input data;

[0066] 5. The link encryption module is deployed on the working node and is responsible for performing data encryption and decryption operations during the streaming data transmission process according to the algorithm instructions issued by the decision-making module to ensure data security;

[0067] 6. The privacy protection module is deployed on the working node and is responsible for protecting sensitive information during the data processing process according to the privacy protection algorithm configuration input by the decision-making module to ensure that privacy is not leaked.

[0068] As an implementation method, the specific steps of the task feature extraction module are as follows:

[0069] (1) Continuously monitor the status of the currently executed streaming data processing task to ensure real-time tracking of the task process;

[0070] (2) Based on the monitoring data, extract the key features of the task, including but not limited to data traffic, processing delay, computing resource utilization rate, data sensitivity, etc.;

[0071] (3) Adopt a dynamically adjusted time window to predict the flow rate change trend through Kalman filtering; within each window period, use the KS test to compare the distribution differences between the current window and the historical window for numerical data; further, construct a feature matrix, calculate the feature weights through the multi-head attention mechanism, and output the dynamically weighted feature vector as the task feature.

[0072] (4) Transmit the extracted task feature data to the decision-making module of the master node through a secure communication channel for subsequent algorithm selection and task scheduling.

[0073] As an implementation method, the specific steps of the decision-making module are as follows:

[0074] (1) Interact with the task feature extraction module on the working node to obtain the feature data of the currently executed streaming data processing task;

[0075] (2) Evaluate the security requirements and privacy protection requirements of the current task according to the task features (such as data traffic, delay requirements, etc.) and the link encryption and privacy protection requirements submitted by the user;

[0076] (3) Based on the analysis results, select the link encryption algorithm (such as AES, RSA) and privacy protection algorithm (such as differential privacy configuration) that match the current task from the preset algorithm library;

[0077] (4) Send the decision results of the selected link encryption algorithm and privacy protection algorithm to the link encryption module and privacy protection module of the working node to perform subsequent encryption and privacy protection operations.

[0078] As an implementation manner, the specific steps of the streaming data processing module are as follows:

[0079] (1) Receive the segmented streaming computing operators input by the master node;

[0080] (2) According to the received operators, perform real-time processing on the input streaming data to generate intermediate results or final outputs;

[0081] (3) During the processing, call the link encryption module to encrypt and transmit the data, and call the privacy protection module to protect the sensitive data to ensure the security and privacy of the data.

[0082] As an implementation manner, the specific steps of the link encryption module are as follows:

[0083] (1) Receive the input from the decision-making module to obtain the currently selected link encryption algorithm;

[0084] (2) Before the data is sent from the worker node, use the specified encryption algorithm (such as AES-256) to encrypt the data to prevent malicious eavesdropping and data leakage;

[0085] (3) At the data receiving end, use the corresponding decryption algorithm to decrypt the encrypted data and restore the original data for subsequent processing.

[0086] As an implementation manner, the specific steps of the privacy protection module are as follows:

[0087] (1) Receive the input from the decision-making module to obtain the currently selected privacy protection algorithm;

[0088] (2) During the streaming data processing, according to the algorithm parameters, add appropriate random noise (such as Laplace noise) to the sensitive data or calculation results to prevent the privacy features from being inferred;

[0089] (3) Check whether the processed data meets the preset privacy protection standards to ensure that malicious parties cannot extract useful information from the data traffic.

[0090] The embodiment of the present invention also correspondingly discloses a trusted streaming data processing method for the smart grid, and its processing flow is as Figure 2 shown, and the specific processing steps are as follows:

[0091] (1) The user submits a streaming data processing task to the master node and provides link encryption and privacy protection requirements;

[0092] (2) After receiving the task, the master node decomposes the streaming data processing task into multiple streaming computing operators and performs task scheduling according to the load conditions of the worker nodes;

[0093] (3) The task feature extraction module on the worker node monitors the task status in real time, extracts task features (such as data traffic, latency requirements, etc.), and reports the feature data to the master node;

[0094] (4) The decision-making module of the master node dynamically selects appropriate link encryption algorithms and privacy protection algorithms according to the task features and user requirements, and sends the decision results to the worker node;

[0095] (5) The streaming data processing module on the worker node receives and executes streaming computing operators, and collaborates with other modules to complete the task;

[0096] (6) During the data transmission process, the link encryption module encrypts and decrypts the data using the selected encryption algorithm according to the decision result to ensure the security of the data transmission process;

[0097] (7) During the data processing process, the privacy protection module adds noise to the sensitive data based on differential privacy technology to ensure that the private data is not leaked;

[0098] (8) After the task is completed, the worker node returns the processing results (encrypted or anonymized data) to the user through the master node.

[0099] The following takes a representative streaming data processing task as an example. The input data is numerical, the feature dimension is 1, and the security and privacy of the data are guaranteed through link encryption and privacy protection technologies. The specific implementation steps are as follows:

[0100] (1) Start the decision-making module on the master node, and at the same time start the task feature extraction module, link encryption module, privacy protection module, and streaming data processing module on the worker node;

[0101] (2) The task feature extraction module on the worker node identifies that the input data is numerical and the feature dimension is 1, and feeds this feature information back to the decision-making module of the master node;

[0102] (3) The decision-making module selects the initial link encryption and privacy protection strategies according to the feedback feature information, and sends the strategies to the worker node; specifically, the link encryption strategy adopts the AES-128 symmetric encryption algorithm, the privacy protection strategy is differential privacy based on the Laplace mechanism, and the initial noise level is set to 1;

[0103] (4) The worker node reads the input data, and the streaming data processing module performs calculation processing on the data, and performs link encryption and privacy protection operations during the processing;

[0104] (5) The task feature extraction module on the worker node continuously monitors the task features, detects whether there are changes, and feeds back the information according to the changes to the master node;

[0105] (6) The system continuously executes steps 2 to 5, dynamically monitors changes in task characteristics, and adjusts the encryption and privacy protection parameters accordingly to adapt to changes in data input, ensuring the credibility and efficiency of the processing process.

[0106] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent replacements made by those of ordinary skill in the art to the technical solutions of the present invention shall all be covered within the protection scope of the present invention, and the protection scope of the present invention shall be subject to that defined by the claims.

Claims

1. A trusted streaming data processing system for the smart grid, comprising the following steps: A master node and multiple worker nodes, where the master node is connected to each worker node through a network; The master node deploys a decision-making module; The worker nodes deploy a task feature extraction module, a streaming data processing module, a link encryption module, and a privacy protection module; The task feature extraction module is used to monitor the status of the streaming data processing task and extract task features based on a dynamically adjusted time window strategy and data stream characteristics; The decision-making module is used to receive the task features, select applicable encryption algorithms and privacy protection algorithms from a preset algorithm library based on the calculated adaptation scores, and send the selected algorithms to each worker node; The streaming data processing module is used to receive the streaming computing operators sent by the master node and perform real-time data processing; The link encryption module is used to encrypt and decrypt data during data transmission according to the algorithm instructions sent by the decision-making module; The privacy protection module is used to add random noise to sensitive data or processing results during the streaming data processing to achieve privacy protection.

2. The trusted streaming data processing system for the smart grid as claimed in claim 1, wherein The task feature extraction module is also used for: Continuously monitoring the execution status of the streaming data processing task on this worker node; Extracting task feature vectors based on real-time streaming data characteristics and task status; Generating a digital signature for the feature vector and uploading it to the master node through a secure channel at a predetermined period.

3. The trusted streaming data processing system for smart grid according to claim 1 or 2, characterized in that, The task feature extraction module is also used for: Adopting a dynamically adjusted time window, where the window size is inversely proportional to the data flow rate, and predicting the flow rate change trend through Kalman filtering; Within each window period, using the KS test for numerical data to compare the distribution differences between the current window and the historical window, and constructing a feature matrix; Calculating feature weights through a multi-head attention mechanism and outputting the dynamically weighted feature vector as the task feature.

4. The trusted streaming data processing system for smart grid according to claim 1, wherein The task features include data traffic, distribution difference, processing delay, computing resource utilization rate, and data sensitivity index.

5. The trusted streaming data processing system for smart grid according to claim 1, characterized in that The decision-making module is also used for: Receiving and verifying the feature vectors and signatures from each worker node; Calculating the adaptation scores of candidate algorithms based on the received feature vectors, the real-time load information of each worker node, and the network latency status; Screening out the link encryption algorithm and privacy protection algorithm with the highest adaptation score from the preset algorithm library, and determining the algorithm identification and parameter configuration; Encapsulating the algorithm identification and parameters into a decision-making instruction, sending it to the corresponding worker node, and at the same time confirming the instruction reception status.

6. The trusted streaming data processing system for the smart grid according to claim 1, wherein The streaming data processing module is also used for: Receiving the streaming computing operators and execution topologies sent by the master node; Sharding and scheduling the input data stream according to the execution topology; Before each operator execution, calling the link encryption module to encrypt or decrypt the data block to be processed; After the operator processing is completed, calling the privacy protection module to add differential privacy noise to sensitive data or intermediate results; Outputting the processed data according to the specified path and reporting the processing status in real time.

7. The trusted streaming data processing system for smart grid according to claim 1, wherein The link encryption module is also used for: Receiving the encryption algorithm identification and key parameters sent by the decision-making module; Obtaining or generating symmetric or asymmetric keys required for encryption / decryption from the key management unit; Encrypt each data packet sent to the network by executing a specified algorithm, and decrypt the encrypted data packet at the receiving end; Perform integrity verification on the decrypted data and feedback the verification result to the streaming data processing module.

8. The trusted streaming data processing system for smart grid according to claim 1, characterized in that, The privacy protection module is further configured to: Receive the privacy protection algorithm identifier and noise parameters issued by the decision-making module; Apply Laplace or Gaussian noise to sensitive fields at each stage of streaming processing; Verify whether the data after adding noise meets the preset privacy budget and differential privacy ε-δ standard; Transmit the privacy-protected data to the downstream module and record the noise usage and privacy budget consumption.

9. A trusted streaming data processing method for a smart grid, which is applied to the trusted data synchronization system for a smart grid described in any one of claims 1-8, and is characterized in that, The method includes the following steps: 1) The master node receives the streaming data processing task submitted by the user and the requirements for link encryption and privacy protection, and decomposes the streaming data processing task into multiple streaming computing operators; 2) The master node issues the streaming computing operators and the requirements for link encryption and privacy protection to the worker nodes; 3) The worker node monitors the execution status of the received streaming computing operators and extracts task characteristics including data traffic, processing delay, computing resource utilization rate, and data sensitivity; 4) The worker node sends the task characteristics to the master node; 5) The master node selects a link encryption algorithm and a privacy protection algorithm from a preset algorithm library based on the task characteristics and the link encryption and privacy protection requirements, and issues the selected algorithm identifier and parameter configuration to the worker node; 6) During the streaming data processing, the worker node calls the link encryption algorithm to encrypt and decrypt the data transmission data packet, and calls the privacy protection algorithm to add random noise to the sensitive data or intermediate result, and returns the processed data to the user through the master node.

Images