Real-time query analysis system and method for electric power data, storage medium and processor

Abstract

The invention relates to the technical field of data processing, in particular to a power data real-time query analysis system and method, a storage medium and a processor, the system comprises a data acquisition module, a data storage module, a sensitive database, a processing module, a data desensitization module, a request analysis module and a query end; according to the method, whether the secret-related level of the data required by the user is higher than the level of the identity ID or not is comprehensively judged according to the identity ID of the user and the sensitive information in the sensitive database, so that a simple and rough mode of taking a person as a judgment standard in the existing power data query requirement is avoided, and the data query request of the identity ID is recorded and analyzed; and the risk of electric power data leakage is reduced. According to the method, the data is subjected to desensitization processing when the data is preliminarily fed back to be confidential, so that the risk of electric power data leakage is further reduced, and the requirement of data query can also be met at the same time.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a real-time query and analysis system, method, storage medium and processor for power data. Background Art

[0002] With the development of smart grid and Internet of Things technologies, the power system has generated a huge amount of data, including key parameters such as voltage, current, power, power quality, and equipment status. In order to extract valuable information from this data to support the stable operation and optimized management of the power grid, a system that can analyze and process big data in real time is needed. This system must be able to handle data collection, cleaning, denoising, format unification, and data security and privacy protection. At present, the security of judging whether the power data required for query is confidential is not high, which is easy to lead to information leakage and insufficient control. Summary of the invention

[0003] In view of the problems in the prior art, the present invention provides a real-time query and analysis system, method, storage medium and processor for power data. The specific technical solutions are as follows:

[0004] A real-time query and analysis system for power data, comprising a data acquisition module, a data storage module, a sensitive database, a processing module, a data desensitization module, a request parsing module, and a query terminal;

[0005] The data acquisition module, data storage module, processing module, data desensitization module, and query end are connected in sequence; the request parsing module is connected to the processing module and the query end respectively; the sensitive database is connected to the processing module;

[0006] The data acquisition module is used to collect power data and send the collected data to the data storage module for storage;

[0007] The sensitive database is used to store sensitive information of power data;

[0008] The query terminal is used for the user to log in with an identity ID and input a power data query request;

[0009] The request parsing module is used to parse the power data query request input by the query end to obtain the request parsing result;

[0010] The processing module is used to retrieve the power data in the data storage module according to the request analysis result to generate preliminary feedback data, and determine whether the preliminary feedback data is higher than the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID. If it is higher, the preliminary feedback data is transmitted to the data desensitization module; if it is not higher, the preliminary feedback data is fed back to the query end;

[0011] The data desensitization module is used to perform desensitization processing on the preliminary feedback data to obtain desensitized feedback data, and feed the desensitized feedback data back to the query end.

[0012] Preferably, the processing module determines whether the preliminary feedback data exceeds the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID, including:

[0013] The processing module matches the preliminary feedback data with the sensitive information in the sensitive database to determine the confidentiality level of the preliminary feedback data;

[0014] The processing module compares the level of authority corresponding to the identity ID with the confidentiality level of the preliminary feedback data. When the confidentiality level of the preliminary feedback data is higher than the confidentiality level that can be viewed corresponding to the identity ID, the preliminary feedback data will be transmitted to the data desensitizing module; otherwise, the preliminary feedback data will be fed back to the query end.

[0015] Preferably, the processing module matches the preliminary feedback data with the sensitive information in the sensitive database, and determines the confidentiality level of the preliminary feedback data as follows:

[0016] The processing module matches the preliminary feedback data with the sensitive information in the sensitive database to obtain an information matching result, and uses the highest confidentiality level involved in the data in the information matching result as the confidentiality level of the preliminary feedback data.

[0017] Preferably, the processing module matches the preliminary feedback data with the sensitive information in the sensitive database to obtain the information matching result as follows:

[0018] Match(A,B)=w1*Model1(A,B)+w2*Model2(A,B);

[0019] When Match(A,B) is greater than the set threshold, it is determined that the data A in the preliminary feedback data matches the sensitive information B of the power data in the sensitive database;

[0020] Among them, Model1 is the first data matching model; w1 is the weight value of the first data matching model; Model2 is the second data matching model; w2 is the weight value of the second data matching model; Model1(A, B) is the result of matching using the first data matching model; Model2(A, B) is the result of matching using the second data matching model.

[0021] Preferably, the first data matching model is a random forest matching model.

[0022] Preferably, the second data matching model is an ANN network matching model.

[0023] Preferably, the weight value of the first data matching model and the weight value of the second data matching model are determined in the following manner:

[0024]

[0025] Among them, J1 is the prediction accuracy of the first data matching model; J2 is the prediction accuracy of the second data matching model.

[0026] A real-time query and analysis method for power data, applied to the system, comprises:

[0027] The data acquisition module collects power data and sends the collected data to the data storage module for storage;

[0028] The sensitive database stores sensitive information of power data;

[0029] Log in to the query terminal with the user ID and enter the power data query request;

[0030] The request parsing module parses the power data query request input by the query end to obtain the request parsing result;

[0031] The processing module retrieves the power data in the data storage module according to the request analysis result to generate preliminary feedback data, and determines whether the preliminary feedback data is higher than the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID. If it is higher, the preliminary feedback data is transmitted to the data desensitization module; if it is not higher, the preliminary feedback data is fed back to the query end;

[0032] The data desensitizing module desensitizes the preliminary feedback data to obtain desensitized feedback data, and feeds the desensitized feedback data back to the query end.

[0033] A computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the real-time query and analysis system for power data.

[0034] A processor is used to run a program, wherein the real-time query and analysis system for power data is executed when the program is running.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention uses the user's identity ID and sensitive information in a sensitive database to comprehensively judge whether the confidentiality level of the data required by the user is higher than the level of the identity ID, avoiding the simple and crude method of using people as the judgment standard in existing power data query requirements, and also records and analyzes data query requests for the identity ID, reducing the risk of power data leakage.

[0037] The present invention performs desensitization processing on the data when the initial feedback data is confidential, thereby further reducing the risk of power data leakage and meeting the needs of data query at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0039] Figure 1 It is a system principle diagram of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0042] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0043] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0044] Embodiment 1:

[0045] like Figure 1 As shown, this embodiment provides a real-time query and analysis system for power data, including a data acquisition module, a data storage module, a sensitive database, a processing module, a data desensitization module, a request parsing module, and a query terminal;

[0046] The data acquisition module, data storage module, processing module, data desensitization module, and query end are connected in sequence; the request parsing module is connected to the processing module and the query end respectively; the sensitive database is connected to the processing module;

[0047] The data acquisition module is used to collect power data and send the collected data to the data storage module for storage;

[0048] The sensitive database is used to store sensitive information of power data;

[0049] The query terminal is used for the user to log in with an identity ID and input a power data query request;

[0050] The request parsing module is used to parse the power data query request input by the query end to obtain the request parsing result;

[0051] The processing module is used to retrieve the power data in the data storage module according to the request analysis result to generate preliminary feedback data, and determine whether the preliminary feedback data is higher than the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID. If it is higher, the preliminary feedback data is transmitted to the data desensitization module; if it is not higher, the preliminary feedback data is fed back to the query end;

[0052] The data desensitization module is used to perform desensitization processing on the preliminary feedback data to obtain desensitized feedback data, and feed the desensitized feedback data back to the query end.

[0053] The data collected by the data acquisition module include but are not limited to voltage, current, power, power quality, equipment information, equipment status, power usage data, etc. The data acquisition module performs preliminary processing on these data, including data cleaning, denoising and format unification, to facilitate subsequent query analysis and processing, and outputs cleaned and formatted power data to provide accurate data for subsequent analysis.

[0054] Data cleaning is a key step that involves preliminary processing of data after it is collected from various data sources in the power system, including removing duplicate data to avoid affecting subsequent analysis, converting data into a unified format for further cleaning and analysis, and handling missing values ​​in the data, which is usually done through interpolation methods such as linear interpolation or polynomial interpolation, or supplementing according to relevant rules to ensure the integrity and accuracy of the data.

[0055] Denoising is the process of identifying and removing outliers or errors from data, which is essential to improving data quality and accuracy. In power big data, denoising usually involves using statistical analysis methods, such as Z-score or IQR, to detect outliers that are significantly different from most of the data, and then processing these outliers, which can be deleted, replaced, or further analyzed to determine their source and nature to ensure the reliability of the data.

[0056] Format unification is the process of ensuring that data is consistent and comparable before analysis. This includes encoding the data to ensure that all data has a unified format and unit, normalizing the data so that it is on the same scale for easy comparison and analysis, and performing language detection and standardization on text data to ensure data consistency. Through these steps, the system can ensure that the input data is clean, accurate and consistent, thereby improving the quality of data analysis and the effectiveness of decision-making.

[0057] The sensitive information of power data stored in sensitive databases involves multiple aspects, including user information, device information, contract information, bill information, power supply information, power market information, etc. User information includes name, ID number, address, contact information, user name and password, user payment information, etc. Power usage data includes power consumption data, load curve, power consumption pattern, etc. Device information includes the unique identifier (ID) of the smart meter, firmware version, installation location, device operating status, fault information, etc.

[0058] Among them, the processing module determines whether the preliminary feedback data exceeds the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID, including:

[0059] The processing module matches the preliminary feedback data with the sensitive information in the sensitive database to determine the confidentiality level of the preliminary feedback data;

[0060] The processing module compares the level of authority corresponding to the identity ID with the confidentiality level of the preliminary feedback data. When the confidentiality level of the preliminary feedback data is higher than the confidentiality level that can be viewed corresponding to the identity ID, the preliminary feedback data will be transmitted to the data desensitizing module; otherwise, the preliminary feedback data will be fed back to the query end.

[0061] The processing module matches the preliminary feedback data with the sensitive information in the sensitive database and determines the confidentiality level of the preliminary feedback data as follows:

[0062] The processing module matches the preliminary feedback data with the sensitive information in the sensitive database to obtain an information matching result, and uses the highest confidentiality level involved in the data in the information matching result as the confidentiality level of the preliminary feedback data.

[0063] The processing module matches the preliminary feedback data with the sensitive information in the sensitive database, and the method for obtaining the information matching result is as follows:

[0064] Match(A,B)=w1*Model1(A,B)+w2*Model2(A,B);

[0065] When Match(A,B) is greater than the set threshold, it is determined that the data A in the preliminary feedback data matches the sensitive information B of the power data in the sensitive database;

[0066] Wherein, Model1 is the first data matching model; w1 is the weight value of the first data matching model; Model2 is the second data matching model; w2 is the weight value of the second data matching model; Model1(A, B) is the result of matching using the first data matching model; Model2(A, B) is the result of matching using the second data matching model. The weight value of the first data matching model and the weight value of the second data matching model are determined in the following way:

[0067]

[0068] Among them, J1 is the prediction accuracy of the first data matching model; J2 is the prediction accuracy of the second data matching model.

[0069] The first data matching model is a random forest matching model, and the second data matching model is an ANN network matching model.

[0070] The training method of the random forest matching model is:

[0071] Data preparation: Collect sensitive information of historical power data, perform preprocessing, and handle missing values, outliers, and duplicate data; use one-hot encoding to convert categorical features into numerical features to form the original data set;

[0072] Random sampling and feature selection: Samples are randomly selected from the original data set with replacement to form multiple training sets. The size of each training set is usually the same as the original data set, but there may be repeated samples. This method ensures that the training data for each decision tree has a certain diversity.

[0073] In random feature selection, at each node splitting process, a subset of features is randomly selected for splitting instead of using all features. For example, in classification problems, the square root of the number of features is often selected as the splitting feature.

[0074] Build a decision tree: For each node, split it using the selected features and splitting criteria (such as information gain or Gini index). Repeat the splitting process recursively until a stopping condition (such as maximum depth or minimum number of samples) is met. Each decision tree starts from the root node and splits recursively until the stopping condition is met. No pruning is performed to keep the growth of each tree to the maximum.

[0075] Matching: Each tree outputs a continuous value, and the final prediction result is the average of all tree prediction values, which is the output result of the random forest matching model.

[0076] Evaluation: Use cross-validation or out-of-bag (OOB) data to evaluate the performance of the model. OOB data refers to samples that were not selected during the Bootstrap sampling process and can be used to estimate the generalization error of the model.

[0077] When the performance of the model meets the requirements, it can be used to determine the matching degree between the preliminary feedback data and the sensitive information.

[0078] The training method of ANN network matching model is:

[0079] Data collection and processing: Collect sensitive information of historical power data, perform preprocessing, and handle missing values, outliers, and duplicate data; use one-hot encoding to convert categorical features into numerical features to form the original data set, and divide the original data set into training set, validation set, and test set;

[0080] Initialization: establishing an ANN network matching model, wherein the ANN network matching model includes an input layer, two hidden layers and an output layer, and using random numbers to initialize the weights and biases of each layer of the ANN network matching model;

[0081] Forward propagation: input the training set and validation set into the initialized ANN network matching model to calculate the output of each neuron; Loss calculation: calculate the value of the loss function to evaluate the gap between the model prediction result and the true value;

[0082] Backpropagation: Calculate the gradient of the loss function with respect to each weight; Update weights and biases using gradient descent;

[0083] Iterative training: Repeat forward propagation, loss calculation and back propagation until the prediction accuracy of the ANN network matching model converges or reaches the preset number of training times, and obtain the trained ANN network matching model;

[0084] Model evaluation: Use the test set to evaluate the trained ANN network matching model, calculate the prediction accuracy and evaluation index, and output the trained ANN network matching model when the prediction accuracy is greater than or equal to the set value and the evaluation index meets the set conditions, otherwise retrain.

[0085] The present invention also includes a data sandbox module, which is connected to the data desensitization module and the query end respectively. Data sandbox privacy computing is a technology for processing data in an isolated environment, which allows data to be analyzed and calculated without being leaked. This technology realizes the value of data fusion by jointly calculating and analyzing data through an encrypted collaboration mechanism.

[0086] The present invention performs desensitization processing on the preliminary feedback data. Data desensitization creates a data copy that looks similar to the original data but does not contain the real sensitive details by strategically modifying or replacing the original data, so as to be safely used in non-production environments such as development, testing, analysis or training. The application of this technology not only protects sensitive information and private data, prevents unauthorized access or leakage, but also meets compliance requirements and reduces the risk of data leakage.

[0087] Embodiment 2:

[0088] Based on the same inventive concept as that of Embodiment 1, this embodiment provides a real-time query and analysis method for power data, which is applied to the system, and includes:

[0089] The data acquisition module collects power data and sends the collected data to the data storage module for storage;

[0090] The sensitive database stores sensitive information of power data;

[0091] Log in to the query terminal with the user ID and enter the power data query request;

[0092] The request parsing module parses the power data query request input by the query end to obtain the request parsing result;

[0093] The processing module retrieves the power data in the data storage module according to the request analysis result to generate preliminary feedback data, and determines whether the preliminary feedback data is higher than the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID. If it is higher, the preliminary feedback data is transmitted to the data desensitization module; if it is not higher, the preliminary feedback data is fed back to the query end;

[0094] The data desensitizing module desensitizes the preliminary feedback data to obtain desensitized feedback data, and feeds the desensitized feedback data back to the query end.

[0095] Embodiment 3:

[0096] Based on the same inventive concept as Example 1, this embodiment provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the real-time query and analysis system for power data.

[0097] Embodiment 4:

[0098] Based on the same inventive concept as that of Embodiment 1, this embodiment provides a processor, which is used to run a program, wherein the real-time query and analysis system for power data is executed when the program is running.

[0099] Those of ordinary skill in the art will appreciate that the modules of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0100] In the embodiments provided by the present invention, it should be understood that the division of modules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules can be combined into one module, one module can be split into multiple modules, or some features can be ignored, etc.

[0101] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0102] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program codes.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A real-time query and analysis system for power data, characterized in that: It includes data collection module, data storage module, sensitive database, processing module, data desensitization module, request parsing module and query terminal; The data acquisition module, data storage module, processing module, data desensitization module, and query end are connected in sequence; the request parsing module is connected to the processing module and the query end respectively; the sensitive database is connected to the processing module; The data acquisition module is used to collect power data and send the collected data to the data storage module for storage; The sensitive database is used to store sensitive information of power data; The query terminal is used for the user to log in with an identity ID and input a power data query request; The request parsing module is used to parse the power data query request input by the query end to obtain the request parsing result; the processing module is used to retrieve the power data in the data storage module according to the request parsing result to generate preliminary feedback data, and judge whether the preliminary feedback data is higher than the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID. If it is higher, the preliminary feedback data is transmitted to the data desensitization module; if it is not higher, the preliminary feedback data is fed back to the query end; The data desensitization module is used to perform desensitization processing on the preliminary feedback data to obtain desensitized feedback data, and feed the desensitized feedback data back to the query end.

2. A real-time query and analysis system for power data according to claim 1, characterized in that: The processing module determines whether the preliminary feedback data exceeds the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID, including: The processing module matches the preliminary feedback data with the sensitive information in the sensitive database to determine the confidentiality level of the preliminary feedback data; The processing module compares the level of authority corresponding to the identity ID with the confidentiality level of the preliminary feedback data. When the confidentiality level of the preliminary feedback data is higher than the confidentiality level that can be viewed corresponding to the identity ID, the preliminary feedback data will be transmitted to the data desensitizing module; otherwise, the preliminary feedback data will be fed back to the query end.

3. A real-time query and analysis system for power data according to claim 2, characterized in that: The processing module matches the preliminary feedback data with the sensitive information in the sensitive database and determines the confidentiality level of the preliminary feedback data as follows: The processing module matches the preliminary feedback data with the sensitive information in the sensitive database to obtain an information matching result, and takes the highest confidentiality level involved in the data in the information matching result as the confidentiality level of the preliminary feedback data.

4. A real-time query and analysis system for power data according to claim 3, characterized in that: The processing module matches the preliminary feedback data with the sensitive information in the sensitive database, and the method for obtaining the information matching result is as follows: Match(A,B)=w1*Model1(A,B)+w2*Model2(A,B); When Match(A,B) is greater than the set threshold, it is determined that the data A in the preliminary feedback data matches the sensitive information B of the power data in the sensitive database; Among them, Model1 is the first data matching model; w1 is the weight value of the first data matching model; Model2 is the second data matching model; w2 is the weight value of the second data matching model; Model1(A, B) is the result of matching using the first data matching model; Model2(A, B) is the result of matching using the second data matching model.

5. A real-time query and analysis system for power data according to claim 4, characterized in that: The first data matching model is a random forest matching model.

6. A real-time query and analysis system for power data according to claim 4, characterized in that: The second data matching model is an ANN network matching model.

7. A real-time query and analysis system for power data according to claim 4, characterized in that: The weight value of the first data matching model and the weight value of the second data matching model are determined in the following manner: Among them, J1 is the prediction accuracy of the first data matching model; J2 is the prediction accuracy of the second data matching model.

8. A real-time query and analysis method for power data, characterized in that: The system applied to any one of claims 1 to 6 comprises: The data acquisition module collects power data and sends the collected data to the data storage module for storage; The sensitive database stores sensitive information of power data; Log in to the query terminal with the user ID and enter the power data query request; The request parsing module parses the power data query request input by the query end to obtain the request parsing result; The processing module retrieves the power data in the data storage module according to the request analysis result to generate preliminary feedback data, and determines whether the preliminary feedback data is higher than the confidentiality level that can be viewed by the identity ID according to the sensitive information in the sensitive database and the authority of the identity ID. If it is higher, the preliminary feedback data is transmitted to the data desensitization module; if it is not higher, the preliminary feedback data is fed back to the query end; The data desensitizing module desensitizes the preliminary feedback data to obtain desensitized feedback data, and feeds the desensitized feedback data back to the query end.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the real-time query and analysis system for power data according to any one of claims 1 to 7.

10. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the real-time query and analysis system for power data according to any one of claims 1 to 7.

Images