A dynamic storage and correction method and system based on meter reading data

Abstract

The present invention relates to the technical field of data storage and correction, and discloses a dynamic storage and correction method and system based on meter reading data, including: obtaining original meter reading data by reading the meter; preprocessing and compressing the original meter reading data through blockchain, and storing the compressed data; integrating the compressed data into a data lake, and performing data analysis using an adversarial network improved by an attention mechanism; and correcting and storing the meter reading data based on the analysis results. Real-time collection of various meter data is achieved, ensuring the integrity and timeliness of the data. Through the application of blockchain technology, efficient and secure storage and management of data is achieved, storage space occupancy is reduced, data transmission efficiency is improved, and the authenticity and integrity of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and correction, and in particular to a dynamic storage and correction method and system based on meter reading data. Background Art

[0002] With the rapid development of Internet of Things technology, smart meters have gradually become popular and widely used in the fields of electricity, water, gas, etc. Smart meters can collect and transmit user usage data in real time through built-in sensors and communication modules, realizing automated and refined management. The traditional meter reading method relies on manual reading and recording, which is not only inefficient, but also prone to reading errors and data loss. Smart meters greatly improve the accuracy and timeliness of data by regularly collecting and transmitting data. In addition, blockchain technology, as a decentralized distributed ledger technology, provides a new solution for data storage and transmission with its high security, immutability and transparency. By storing meter reading data in the blockchain, the integrity and security of the data can be ensured. However, despite the significant advantages of smart meters and blockchain technology in meter reading data management, there are still some technical challenges that need to be solved.

[0003] In the current smart meter data management system, the efficiency of data processing and storage and the accuracy of data are two major technical challenges. First, the amount of data generated by smart meters is huge, and how to efficiently preprocess, compress and store this data has become a key issue. The existing methods for data preprocessing and compression are relatively simple, and it is often difficult to achieve efficient storage and transmission while ensuring data integrity. Secondly, although blockchain technology provides data immutability, data errors or anomalies may still occur during data upload. The existing anomaly detection and data correction methods are relatively simple, usually relying only on simple rules or models, and it is difficult to accurately identify and correct complex abnormal data. Especially when faced with multi-source heterogeneous data, the existing technology seems to be unable to cope with data integration and analysis, and it is difficult to effectively utilize the time dimension and device dimension characteristics of the data, thereby affecting the accuracy of data analysis and the reliability of decision-making. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing dynamic storage and correction methods for meter reading data have the problems of low data processing efficiency, insufficient accuracy in abnormal data detection and correction, etc.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a dynamic storage and correction method based on meter reading data, comprising:

[0007] By reading the meter, the original meter reading data is obtained;

[0008] Preprocessing and compressing the original meter reading data through blockchain, and storing the compressed data;

[0009] Integrate the compressed data into the data lake and use the adversarial network improved by the attention mechanism to perform data analysis;

[0010] Based on the analysis results, the meter reading data is corrected and stored.

[0011] As a preferred solution of the method for dynamic storage and correction based on meter reading data described in the present invention, wherein: the original meter reading data includes meter reading data collection for intelligent meters and non-intelligent meters;

[0012] For the smart meter, meter reading data is automatically collected and transmitted at regular intervals, and a timestamp is matched for each meter reading data;

[0013] For non-intelligent meters, the meters are read through sensors, and after recording the timestamp of the reading time, the meter reading data is uploaded through Internet devices.

[0014] As a preferred solution of the method for dynamic storage and correction based on meter reading data described in the present invention, wherein: the blockchain includes that the device sends data to the blockchain node through the network of the Internet of Things;

[0015] The meter reading data is pre-processed and compressed in the blockchain node, the compressed data is packaged into blocks, and the blocks are added to the blockchain through a consensus mechanism;

[0016] The preprocessing includes removing redundant and invalid parts in the data and converting the data into a standard format.

[0017] As a preferred solution of the dynamic storage and correction method based on meter reading data described in the present invention, wherein: the integration of the compressed data into the data lake includes extracting data information of each block from the blockchain, decompressing the extracted compressed data, and restoring the data;

[0018] Align data by timestamp to keep data in different blocks consistent in time dimension as the first data lake;

[0019] Classify the data in the block according to the meter device number so that the data in different meters are consistent in the device dimension, which serves as the second data lake;

[0020] The improved adversarial network is used to perform data analysis on the first data lake and the second data lake respectively.

[0021] As a preferred solution of the dynamic storage and correction method based on meter reading data described in the present invention, the attention mechanism includes constructing an attention mechanism for the first data lake in the time dimension, and the specific steps are:

[0022] Perform time series embedding on the meter data and extract time dimension features;

[0023] ,

[0024] in, represents the i-th vector of the time series embedding matrix, and TemporalEmbedding represents the encoder of time series embedding;

[0025] By introducing external factors, an external factor matrix is ​​constructed;

[0026] ,

[0027] Among them, E represents the external factor matrix, represents the mth external factor;

[0028] Using Bayesian inference and wavelet transform for feature extraction and weighting of time series and external factors;

[0029] By analyzing the probability distribution of meter data in the time dimension through Bayesian inference, the posterior distribution is updated in combination with external factors;

[0030] ,

[0031] Extract multi-scale features from time series through wavelet transform;

[0032] ,

[0033] Combining Bayesian inference and wavelet transform to improve the attention mechanism;

[0034] ,

[0035] ,

[0036] ,

[0037] in, represents the posterior probability of the meter data X under the external factor E; represents the likelihood of external factor E under the condition of meter data X; represents the prior probability of the meter data X; represents the marginal probability of the external factor E; Wavelet transform coefficients representing meter data; represents the conjugate complex number of the mother wavelet; a represents the scale parameter; b represents the translation parameter; Indicates the abnormality assessment result for the first data lake. A value of 1 indicates a serious abnormality, and a value of 0 indicates no abnormality. represents the dimension normalization factor of the key matrix in the first data lake, represents the dimension of the time series embedding matrix, represents the dimension of the external factor matrix E; softmax represents the normalized exponential function; t represents the time variable at the time point sampled in the time series; V represents the value matrix.

[0038] As a preferred solution of the dynamic storage and correction method based on meter reading data described in the present invention, the attention mechanism also includes constructing an attention mechanism for the second data lake in the device dimension, and the specific steps are:

[0039] Embed the device usage information into the data of a single device and extract the device usage characteristics;

[0040] ,

[0041] in, represents the usage data of device j, represents the embedding vector of device j, and UsageEmbedding represents the encoder of device usage embedding;

[0042] Introducing an adaptive weight matrix to dynamically adjust weights based on device usage;

[0043] ,

[0044] Among them, U indicates that the device uses an embedded vector set; AdaptiveWeightMatrix indicates a dynamically adjusted weight function;

[0045] Kalman filtering and multi-scale decomposition are used to extract and weight features of single device data:

[0046] Use Kalman filtering to estimate the state of device data and update the estimate based on usage;

[0047] ,

[0048] ,

[0049] in, is the state estimate of the Kalman filter, which represents the state estimate after the observation data is given at time t; is the predicted state estimate of the Kalman filter, which represents the estimated state value at time t predicted based on the previous state and observation data at time t−1; is the Kalman gain matrix, used to update the state estimate; represents the observed value at time t; Represents the observation model matrix, which is used to map variables in the state space to variables in the observation space; represents the prediction error covariance matrix, which is used to describe the uncertainty of the state estimate at time t predicted at time t−1; Represents the observation model matrix The transpose of represents the observation noise covariance matrix;

[0050] Extract multi-scale features of device data using multi-scale decomposition;

[0051] ,

[0052] Among them, M(x) represents the multi-scale decomposition feature, which is used to extract the multi-scale features of the data; represents the j-th level decomposition feature; J represents the number of scales in the multi-scale decomposition; j represents the scale index; x represents the input data, which is the updated data;

[0053] Combine Kalman filtering and multi-scale decomposition to improve the attention mechanism;

[0054] ,

[0055] ,

[0056] ,

[0057] in, represents the dimension normalization factor of the key matrix in the second data lake, Indicates the dimensions of the embedding matrix used by the device, represents the dimension of the adaptive weight matrix; Indicates the anomaly assessment result for the second data lake. A value of 1 indicates a serious anomaly, and a value of 0 indicates no anomaly.

[0058] As a preferred solution of the dynamic storage and correction method based on meter reading data described in the present invention, wherein: the adversarial network includes using historical data to train the generator G and the discriminator D, the generator G learns to generate realistic data, and the discriminator D learns to distinguish between real data and generated data;

[0059] Generate candidate correction data through the generator , and use the discriminator and attention mechanism of the first data lake and the second data lake to conduct confrontation, so as to select the most likely corrected data and apply it to the real data;

[0060] Generator objective function:

[0061] ,

[0062] in, Represents the loss function that minimizes the generator G , represents the loss function of the generator, E represents the expected value operation, Represents the noise distribution The noise z sampled in The candidate correction data generated by the generator G for the noise z;

[0063] Discriminator objective function:

[0064] max D L D = E x∼ p data x [logD I A F x,T,E +logD( I A S (x,U,A) + E z∼ p z (z) [ log (1-D( I A F (G(z),T ,E)))+ log (1-D( I A S (G(z),T ,E))] ,

[0065] in, Represents the loss function that maximizes the generator D , represents the loss function of the discriminator, Represents the distribution from real data The real data x sampled in, D(...) represents the score of the discriminator D on the data;

[0066] Comprehensive confrontation formula:

[0067] L GAN = min G m D [ E x∼ p data (x) [ log D( I A F (x,T ,E))+ log D( I A S (x,U,A))+ E z∼ p z (z) [ log (1-D( I A F (G(z),T ,E)))+ log (1-D( I A S (G(z),T ,E))] ,

[0068] in, Represents the comprehensive adversarial loss function of the generative adversarial network;

[0069] The specific steps for data correction are:

[0070] For input data x, the probability D(x) output by the discriminator D is used to evaluate the authenticity of the data x; if D(x) is lower than the threshold τ, the data x needs to be corrected; for each data x that needs to be corrected, multiple candidate corrected data are generated by the generator G:

[0071] ,

[0072] Where N represents the number of data that needs to be corrected. Denotes the noise of the generator G Generated candidate correction data, Represents the standard normal distribution The noise sampled , i represents the index;

[0073] The generated candidate correction data are scored by the discriminator D and the attention mechanism results, and the data with the highest score is selected As a result of the correction;

[0074] x corrected =arg max G z i D I A F G z i ,T,E +D( I A S (G z i , U , A)] ,

[0075] Among them, argmax means taking the value that makes the expression reach the maximum value. ;

[0076] After completing the data correction, the meter reading data is corrected and stored; when displaying the data, the corrected data is displayed first, and the original data can be viewed through tracing.

[0077] On the other hand, a dynamic storage and correction system based on meter reading data using the method of the present invention, wherein:

[0078] The collection unit obtains the original meter reading data by reading the meter;

[0079] A transmission unit, which preprocesses and compresses the original meter reading data through blockchain, and stores the compressed data;

[0080] The analysis unit integrates the compressed data into the data lake and uses the adversarial network improved by the attention mechanism to perform data analysis;

[0081] The storage unit corrects and stores the meter reading data according to the analysis result.

[0082] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein: the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0083] A computer-readable storage medium stores a computer program, wherein: when the computer program is executed by a processor, the steps of any one of the methods of the present invention are implemented.

[0084] Beneficial effects of the present invention: The dynamic storage and correction method based on meter reading data provided by the present invention realizes the real-time collection of various types of meter data, ensuring the integrity and timeliness of the data. Through the application of blockchain technology, efficient and secure storage and management of data is achieved, storage space occupancy is reduced, data transmission efficiency is improved, and the authenticity and integrity of the data are ensured. Through the integration of data lakes and the analysis of improved adversarial networks, the accuracy and efficiency of data processing and analysis are improved, and anomalies in data can be more accurately identified and corrected to ensure high data quality. Intelligent correction of anomalies and errors in data is achieved, which significantly improves the accuracy and reliability of data and ensures the correctness of subsequent data analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0086] Figure 1 The present invention provides an overall flow chart of a method for dynamic storage and correction of meter reading data according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0088] Example 1, reference Figure 1 , is an embodiment of the present invention, and provides a dynamic storage and correction method based on meter reading data, comprising:

[0089] S1: Obtain original meter reading data by reading the meter.

[0090] Furthermore, the original meter reading data includes meter reading data collection for smart meters and non-intelligent meters. For the smart meters, meter reading data is automatically collected and transmitted at regular intervals, and each meter reading data is matched with a timestamp; for the non-intelligent meters, the meters are read by sensors, and the timestamp of the meter reading time is recorded, and the meter reading data is uploaded through an Internet device.

[0091] It should be said that smart meters have automatic collection functions and collect consumption data regularly. Use embedded systems and sensors to monitor consumption in real time. The interval of regular collection can be set as needed (such as every hour, every day, etc.). Use photoelectric sensors, ultrasonic sensors, etc. to read the values ​​of non-smart meters. Sensors can be installed on the display part of the meter to automatically read the values. The comprehensiveness and accuracy of the data are ensured by collecting meter reading data by different methods for smart meters and non-smart meters. Smart meters ensure the timeliness and continuity of data through automatic collection and transmission; non-smart meters realize automatic data collection and upload through the combination of sensors and Internet devices. These data provide basic support for subsequent anomaly detection and data correction.

[0092] S2: Preprocess and compress the original meter reading data through blockchain, and store the compressed data.

[0093] Furthermore, the device sends data to the blockchain node through the Internet of Things network; the meter reading data is preprocessed and compressed at the blockchain node, the compressed data is packaged into blocks, and the blocks are added to the blockchain through a consensus mechanism. The preprocessing includes removing redundant and invalid parts of the data; converting the data format to a standard format.

[0094] Extract the data information of each block from the blockchain, decompress the extracted compressed data, and restore the data. Align the data by timestamp to keep the data in different blocks consistent in the time dimension, as the first data lake. Classify the data in the block according to the meter device number, so that the data in different meters are consistent in the device dimension, as the second data lake.

[0095] It is important to know that the immutability of blockchain can be used to ensure the security of meter reading data during transmission and storage to prevent data tampering. The integrity of meter reading data can be ensured through the consensus mechanism of blockchain to prevent data loss or damage. Through data preprocessing, compression and decompression, the consistency of meter reading data can be ensured to provide high-quality data for subsequent data analysis and processing. Through data compression, the cost of data storage and transmission can be reduced and efficiency can be improved. Through time alignment and device classification, two data lakes (time dimension and device dimension) are constructed to provide a basis for anomaly detection and data correction.

[0096] The security, integrity and consistency of meter reading data can be ensured from collection, transmission, storage, preprocessing and compression to storage and restoration in the blockchain. Through time alignment and device classification, two data lakes are built, providing a reliable data foundation for subsequent anomaly detection and data correction. Not only does it improve the efficiency and accuracy of data processing, but it also uses blockchain technology to enhance data security and transparency.

[0097] The improved adversarial network is used to perform data analysis on the first data lake and the second data lake respectively.

[0098] S3: Integrate the compressed data into the data lake and use the adversarial network improved by the attention mechanism to perform data analysis.

[0099] Furthermore, in the time dimension, an attention mechanism is constructed for the first data lake. The specific steps are as follows:

[0100] Perform time series embedding on the electric meter data and extract time dimension features.

[0101] ,

[0102] in, represents the i-th vector of the time series embedding matrix, and TemporalEmbedding represents the encoder of time series embedding. The specific encoding formula is:

[0103] ,

[0104] ,

[0105] in, is the time step, is the dimension of the embedding vector, is the index of the embedding dimension.

[0106] Ensuring that different time steps have unique embedding representations enables the model to capture temporal dependencies.

[0107] Assuming the time step The embedding dimension is 4 (i.e. ), then the embedding vector is calculated as follows:

[0108]

[0109] ,

[0110] By introducing external factors, an external factor matrix is ​​constructed.

[0111] ,

[0112] Among them, E represents the external factor matrix, Represents the mth external factor.

[0113] Bayesian inference and wavelet transform are used to extract and weight the features of time series and external factors. The probability distribution of meter data in the time dimension is analyzed by Bayesian inference, and the posterior distribution is updated in combination with external factors.

[0114] ,

[0115] Extracting multi-scale features from time series through wavelet transform.

[0116] ,

[0117] Combining Bayesian inference and wavelet transform to improve attention mechanism.

[0118] ,

[0119] ,

[0120] ,

[0121] in, represents the posterior probability of the meter data X under the external factor E; represents the likelihood of external factor E under the condition of meter data X; represents the prior probability of the meter data X; represents the marginal probability of the external factor E; Wavelet transform coefficients representing meter data; represents the conjugate complex number of the mother wavelet; a represents the scale parameter; b represents the translation parameter; Indicates the abnormality assessment result for the first data lake. A value of 1 indicates a serious abnormality, and a value of 0 indicates no abnormality. represents the dimension normalization factor of the key matrix in the first data lake, represents the dimension of the time series embedding matrix, represents the dimension of the external factor matrix E; softmax represents the normalized exponential function; t represents the time variable at the time point sampled in the time series; V represents the value matrix.

[0122] It is important to know that the dimension of the time series embedding matrix represents how many dimensional vectors each time step is embedded into; if there are many complex patterns in the time series, a higher embedding dimension may be required. Higher embedding dimensions usually improve model performance, but also increase computational cost and the risk of overfitting. If we process a year of meter data, sampled by day, and choose the embedding dimension as 128, then the dimension of the time series embedding matrix is ​​365×128. The dimension of the external factors matrix represents the dimension of each external factor after embedding; generally speaking, the feature dimension of the external factors should be high enough to capture information, but not too high to increase the computational burden. Suppose we consider the following three external factors: temperature, humidity, wind speed. The embedding dimension of each external factor is 64, so the dimension of the external factors matrix is ​​3×64. Suppose we process a year of meter data, sampled by day, and choose the time series embedding dimension as 128. There are three external factors (temperature, humidity, wind speed), and each external factor is embedded with a dimension of 64. The dimension of the time series embedding matrix is ​​365×128. The dimension of the external factors matrix is ​​3×64.

[0123] What needs to be said is that by extracting time dimension features through time series embedding, the model can better understand and process time series data. Constructing an external factor matrix enables the model to comprehensively consider the impact of external factors on meter data and improve the accuracy of anomaly detection. For example, in certain weather conditions, the meter may experience abnormal jumps or abnormal changes in electricity consumption. The probability distribution of data in the time dimension is analyzed through Bayesian inference, and the posterior distribution is updated in combination with external factors; multi-scale features in the time series are extracted through wavelet transform, so that the model can capture data changes at different scales. Combining Bayesian inference and wavelet transform, the attention mechanism is improved so that it can more effectively identify and weight important features, thereby improving the effect of anomaly detection.

[0124] Furthermore, in the device dimension, an attention mechanism is constructed for the second data lake. The specific steps are: embedding the device usage data of a single device and extracting the device usage features.

[0125] ,

[0126] in, represents the usage data of device j, represents the embedding vector of device j, and UsageEmbedding represents the encoder of device usage embedding.

[0127] An adaptive weight matrix is ​​introduced to dynamically adjust the weights according to device usage.

[0128] ,

[0129] Among them, U indicates that the device uses an embedded vector set; AdaptiveWeightMatrix indicates a dynamically adjusted weight function.

[0130] Kalman filtering and multi-scale decomposition are used to extract and weight the features of single device data. Kalman filtering is used to estimate the state of device data and update the estimate based on usage.

[0131] ,

[0132] ,

[0133] in, is the state estimate of the Kalman filter, which represents the state estimate after the observation data is given at time t; is the predicted state estimate of the Kalman filter, which represents the estimated state value at time t predicted based on the previous state and observation data at time t−1; is the Kalman gain matrix, used to update the state estimate; represents the observed value at time t; Represents the observation model matrix, which is used to map variables in the state space to variables in the observation space; represents the prediction error covariance matrix, which is used to describe the uncertainty of the state estimate at time t predicted at time t−1; Represents the observation model matrix The transpose of represents the observation noise covariance matrix.

[0134] Multi-scale decomposition is used to extract multi-scale features of device data.

[0135] ,

[0136] Among them, M(x) represents the multi-scale decomposition feature, which is used to extract the multi-scale features of the data; represents the j-th level decomposition feature; J represents the number of scales in the multi-scale decomposition; j represents the scale index; x represents the input data, which is the updated data.

[0137] Combine Kalman filtering and multi-scale decomposition to improve the attention mechanism.

[0138] ,

[0139] ,

[0140] ,

[0141] in, represents the dimension normalization factor of the key matrix in the second data lake, Indicates the dimensions of the embedding matrix used by the device, represents the dimension of the adaptive weight matrix; Indicates the anomaly assessment result for the second data lake. A value of 1 indicates a serious anomaly, and a value of 0 indicates no anomaly.

[0142] What needs to be said is that through device usage embedding, the usage characteristics of the device are extracted, so that the model can better understand the usage patterns and behaviors of the device. An adaptive weight matrix is ​​introduced to dynamically adjust the weights according to the device usage to better capture the dynamic changes in device usage. Kalman filtering is used to estimate the state of device data, and the state estimation is updated according to the usage to ensure the accuracy of the device state. Multi-scale decomposition is used to extract multi-scale features of device data, so that the model can capture data changes at different scales. Combining Kalman filtering and multi-scale decomposition, the attention mechanism is improved so that it can more effectively identify and weight important features, thereby improving the effect of anomaly detection.

[0143] It is important to know that the device usage embedding encoder includes, embedding matrix initialization: define an embedding matrix and a bias vector . The standardized device usage data Linear transformation is performed via an embedding matrix and a bias vector.

[0144] ,

[0145] Among them, ReLU is the activation function, is the embedding matrix, is the bias vector.

[0146] The dynamic adjustment weight function includes defining a weight matrix and a bias vector . The device uses the embedded vector Perform a linear transformation via a weight matrix and a bias vector.

[0147] ,

[0148] Among them, softmax is a normalized exponential function, is the weight matrix, is the bias vector.

[0149] The generator G and discriminator D are trained using historical data. The generator G learns to generate realistic data, and the discriminator D learns to distinguish between real data and generated data. , and use the discriminator and attention mechanism of the first data lake and the second data lake for confrontation, so as to select the most likely corrected data and apply it to the real data.

[0150] Generator network structure:

[0151] ,

[0152] Discriminator network structure:

[0153] ,

[0154] From the standard normal distribution A random noise vector sampled from is input to the generator network. is the output of the first hidden layer of the generator network, and the calculation formula is ,in is the weight matrix of the first layer, is the bias vector of the first layer and ReLU is the activation function. is the second hidden layer output of the generator network, calculated by ReLU ,in is the weight matrix of the second layer, is the bias vector of the second layer. is the output of the third hidden layer of the generator network, and the calculation formula is ReLU ,in is the weight matrix of the third layer, is the bias vector of the third layer. is the output of the generator network, and the calculation formula is ,in is the weight matrix of the fourth layer, is the bias vector of the fourth layer, is the hyperbolic tangent activation function, which is used to limit the output data range to [-1,1]. The input of the discriminator network can be real data or data generated by the generator. . is the first hidden layer output of the discriminator network, and the calculation formula is LeakyReLU ,in is the weight matrix of the first layer, is the bias vector of the first layer, and LeakyReLU is a leaky linear rectification activation function used to avoid the dead neuron problem of ReLU. is the second hidden layer output of the discriminator network, and the calculation formula is LeakyReLU ),in is the weight matrix of the second layer, is the bias vector of the second layer. is the third hidden layer output of the discriminator network, and the calculation formula is LeakyReLU ),in is the weight matrix of the third layer, is the bias vector of the third layer. is the output of the discriminator network, and the calculation formula is ,in is the weight matrix of the fourth layer, is the bias vector of the fourth layer, It is a sigmoid activation function, which is used to limit the output data range to [0,1], indicating the probability that the input data is true data.

[0155] Generator objective function:

[0156] ,

[0157] in, Represents the loss function that minimizes the generator G , represents the loss function of the generator, E represents the expected value operation, Represents the noise distribution The noise z sampled in The candidate correction data generated by the generator G for the noise z.

[0158] Discriminator objective function:

[0159] max D L D = E x∼ p data x [logD I A F x,T,E +logD( I A S (x,U,A) + E z∼ p z (z) [ log (1-D( I A F (G(z),T ,E)))+ log (1-D( I A S (G(z),T ,E))] ,

[0160] in, Represents the loss function that maximizes the generator D , represents the loss function of the discriminator, Represents the distribution from real data The real data x sampled in , D(...) represents the score of the discriminator D on the data.

[0161] Comprehensive confrontation formula:

[0162] L GAN = min G m D [ E x∼ p data (x) [ log D( I A F (x,T ,E))+ log D( I A S (x,U,A))+ E z∼ p z (z) [ log (1-D( I A F (G(z),T ,E)))+ log (1-D( I A S (G(z),T ,E))] ,

[0163] in, Represents the comprehensive adversarial loss function of the generative adversarial network.

[0164] The specific steps for data correction are:

[0165] For input data x, the probability D(x) output by the discriminator D is used to evaluate the authenticity of the data x; if D(x) is lower than the threshold τ, the data x needs to be corrected; for each data x that needs to be corrected, multiple candidate corrected data are generated by the generator G:

[0166] ,

[0167] Where N represents the number of data that needs to be corrected. Denotes the noise of the generator G Generated candidate correction data, Represents the standard normal distribution The noise sampled , i represents the index.

[0168] The generated candidate correction data are scored by the discriminator D and the attention mechanism results, and the data with the highest score is selected As a result of the correction.

[0169] x corrected =arg max G z i D I A F G z i ,T,E +D( I A S (G z i , U , A)] ,

[0170] Among them, argmax means taking the value that makes the expression reach the maximum value. .

[0171] The discriminator scores the input data to identify low-quality data that may be abnormal. The generator is used to generate multiple candidate correction data, and the discriminator and attention mechanism are used to select the optimal correction data to replace the original abnormal data. The attention mechanism is used to enhance the weighting of features of different dimensions to further improve the accuracy of abnormal data detection and correction. An automated data detection and correction process is implemented to reduce manual intervention and improve efficiency. Combining the generative adversarial network and the attention mechanism, the correction of real data is achieved by generating and selecting the optimal candidate correction data. This method can comprehensively consider multiple factors and improve the accuracy and quality of data correction.

[0172] S4: According to the analysis results, the meter reading data is corrected and stored.

[0173] After completing the data correction, the meter reading data is corrected and stored; when displaying the data, the corrected data is displayed first, and the original data can be viewed through tracing.

[0174] Furthermore, when users query or the system analyzes, the corrected data is displayed by default. Provide an interface or function to allow users or the system to view the original data and its metadata as needed, and trace the data through the associated identifier. Display all version data during the correction process, including the timestamp of each correction, the discriminator score and other information, to ensure the transparency of the correction process.

[0175] On the other hand, this embodiment also provides a dynamic storage and correction system based on meter reading data, which includes:

[0176] The acquisition unit obtains the original meter reading data by reading the meter.

[0177] The transmission unit preprocesses and compresses the original meter reading data through the blockchain, and stores the compressed data.

[0178] The analysis unit integrates the compressed data into the data lake and uses the adversarial network improved by the attention mechanism to perform data analysis.

[0179] The storage unit corrects and stores the meter reading data according to the analysis result.

[0180] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0182] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0183] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0184] Example 2: The following is an embodiment of the present invention, which provides a dynamic storage and correction method based on meter reading data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0185] Experiment 1: Comparison of storage speed and data tracing query speed. The test object is the data of smart meters and non-smart meters in a city's power system, and the collection scope covers the main residential and commercial users in the city. The experiment is divided into two parts: storage speed test and data tracing query speed test. Record the data in Table 1.

[0186] Storage speed test:

[0187] Experimental group: Using the method of the present invention, the original meter reading data was preprocessed and compressed through blockchain, and the compressed data was stored in the blockchain node.

[0188] Control group: Use the traditional hand-copying method to manually record meter reading data into paper files and enter them into electronic spreadsheets for storage.

[0189] During the experiment, the time from collection to storage completion of each set of data was recorded.

[0190] Data traceability query speed test:

[0191] Experimental group: Extract the data information of each block from the blockchain, decompress and restore the data, and query the data based on the timestamp and device number.

[0192] Control group: Corresponding data were manually searched from paper files and electronic forms.

[0193] During the experiment, the time from the start of the query to the acquisition of complete data for each set of data was recorded.

[0194] Table 1 Experimental data table

[0195] Parameter name Experimental Group (Blockchain) Control group (handwritten) Storage speed (seconds) 12 45 Data tracing query speed (s) 5 30 Data loss during storage (%) 0 15 Data tampering Not supported support Data integrity (%) 100 91 Security high Low

[0196] Experiment 2: Comparison of correction accuracy and other advantages. The experimental objects are also the data of smart meters and non-smart meters in a certain city's power system. The experiment is divided into two parts: data correction accuracy test and other advantage test. The data is recorded in Table 2.

[0197] Data Correction Accuracy Test:

[0198] Experimental group: Using the method of the present invention, candidate correction data was generated by generative adversarial network, and scored by discriminator and attention mechanism, and the data with the highest score was selected as the correction result.

[0199] Control group: Use traditional manual methods to manually judge and correct errors in the data.

[0200] During the experiment, the difference before and after correction of each set of data was recorded and compared with the true value to calculate the correction accuracy.

[0201] Other Advantages Tests:

[0202] Experimental group: evaluate the performance of the present invention in terms of data processing speed, data consistency, anomaly detection capability, etc.

[0203] Control group: Evaluate the performance of traditional manual methods on the same indicators.

[0204] During the experiment, the performance of each group in terms of processing speed, data consistency, and anomaly detection capability was recorded.

[0205] Table 2 Performance test data table

[0206] Parameter name Experimental group (the present invention) Control group (manual) Correction accuracy (%) 98 85 Data processing speed (seconds / 1000 data) 8 50 Data consistency (%) 98 89 Anomaly detection capabilities powerful weak Data integrity (%) 100 89 User satisfaction (points) 9.5 6

[0207] The data table of Experiment 1 in Table 1 clearly shows the significant advantages of the dynamic storage method based on blockchain in storage speed and data traceability query speed. The experimental group only takes 12 seconds for storage speed, while the control group takes 45 seconds, indicating that the method of the present invention is efficient in data processing and storage. In terms of data traceability query speed, the experimental group only takes 5 seconds to complete, while the control group takes 30 seconds, further verifying the superiority of blockchain technology in data management. In addition, the experimental group performed well in data loss detection, tampering detection, data integrity and security, showing the significant improvement of the present invention in data reliability and security.

[0208] The superiority of the present invention is further demonstrated by comparing the correction accuracy and other advantages through the data table of Experiment 2 in Table 2. The correction accuracy of the experimental group reached 98%, which is significantly higher than 85% of the control group, indicating the high accuracy and reliability of the method based on the generative adversarial network in data correction. In addition, in terms of data processing speed, the experimental group only takes 8 seconds to process 1,000 data, while the control group takes 50 seconds, showing the advantage of the present invention in data processing efficiency. In terms of data consistency and anomaly detection capabilities, the experimental groups all showed a high level, further verifying the powerful functions of the present invention in data management and anomaly handling. The user satisfaction score also shows that the user satisfaction of the experimental group is significantly higher than that of the control group, reaching 9.5 points, reflecting the efficiency and user recognition of the present invention in practical applications.

[0209] The significant advantages of the present invention in data storage speed, data traceability query speed, data correction accuracy and other aspects are fully demonstrated. Compared with the traditional hand-copying and manual correction methods, the present invention not only improves the efficiency of data processing and management, but also performs well in data security, reliability and user satisfaction, and provides an innovative solution for the application of smart meters and blockchain technology in the field of Internet of Things and big data processing.

[0210] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A dynamic storage and correction method based on meter reading data, characterized in that: include: By reading the meter, the original meter reading data is obtained; Preprocessing and compressing the original meter reading data through blockchain, and storing the compressed data; Integrate the compressed data into the data lake and use the adversarial network improved by the attention mechanism to perform data analysis; According to the analysis results, the meter reading data is corrected and stored; The integrating the compressed data into the data lake includes extracting data information of each block from the blockchain, decompressing the extracted compressed data, and restoring the data; Align data by timestamp to keep data in different blocks consistent in time dimension as the first data lake; Classify the data in the block according to the meter device number so that the data in different meters are consistent in the device dimension, which serves as the second data lake; Using the improved adversarial network, respectively perform data analysis on the first data lake and the second data lake; The adversarial network includes using historical data to train a generator G and a discriminator D, where the generator G learns to generate realistic data and the discriminator D learns to distinguish between real data and generated data; Generate candidate correction data G(z) through the generator i ), and use the discriminator and attention mechanism of the first data lake and the second data lake to conduct adversarial training, so as to select the most likely corrected data and apply it to the real data.

2. The method for dynamic storage and correction based on meter reading data according to claim 1, characterized in that: The original meter reading data includes meter reading data collected for smart meters and non-intelligent meters; For the smart meter, meter reading data is automatically collected and transmitted at regular intervals, and a timestamp is matched for each meter reading data; For non-intelligent meters, the meters are read through sensors, and after recording the timestamp of the reading time, the meter reading data is uploaded through Internet devices.

3. The method for dynamic storage and correction based on meter reading data according to claim 2, characterized in that: The blockchain includes that the device sends data to the blockchain node through the network of the Internet of Things; The meter reading data is pre-processed and compressed in the blockchain node, the compressed data is packaged into blocks, and the blocks are added to the blockchain through a consensus mechanism; The preprocessing includes removing redundant and invalid parts in the data and converting the data into a standard format.

4. The method for dynamic storage and correction based on meter reading data as claimed in claim 3, characterized in that: The attention mechanism includes constructing an attention mechanism for the first data lake in the time dimension, and the specific steps are as follows: Perform time series embedding on the meter data and extract time dimension features; T i =TemporalEmbedding(t i ) Among them, T i represents the i-th vector of the time series embedding matrix, TemporalEmbedding represents the encoder of time series embedding; t i represents the i-th time variable of the encoder input; By introducing external factors, an external factor matrix is ​​constructed; E=[e1,e2,…,e m ] Among them, E represents the external factor matrix, e m represents the mth external factor; Using Bayesian inference and wavelet transform, feature extraction and weighting of time series and external factors are performed; Through Bayesian inference, the probability distribution of meter data in the time dimension is analyzed, and the posterior distribution is updated in combination with external factors; Extract multi-scale features from time series through wavelet transform; Combining Bayesian inference and wavelet transform to improve the attention mechanism; V1=XW1 Among them, P(X|E) represents the posterior probability of the meter data X under the condition of external factor E; P(E|X) represents the likelihood of external factor E under the condition of meter data X; P(X) represents the prior probability of meter data X; P(E) represents the marginal probability of external factor E; Represents the wavelet transform coefficient of the meter data; ψ * represents the conjugate complex number of the mother wavelet; a represents the scale parameter; b represents the translation parameter; IA F (X, T, E) represents the anomaly assessment result for the first data lake, where a value of 1 indicates a severe anomaly and a value of 0 indicates no anomaly; d k1 =d t +d e represents the dimension normalization factor of the key matrix in the first data lake, d t represents the dimension of the time series embedding matrix, d e represents the dimension of the external factor matrix E; softmax represents the normalized exponential function; t represents the time variable at the time point sampled in the time series; V represents the value matrix.

5. The method for dynamic storage and correction based on meter reading data according to claim 4, characterized in that: The attention mechanism also includes constructing an attention mechanism for the second data lake in the device dimension, and the specific steps are: Embed the device usage information into the data of a single device and extract the device usage characteristics; U j =UsageEmbedding(u j ) Among them, u j Represents the usage data of device j, U j represents the embedding vector of device j, and UsageEmbedding represents the encoder of device usage embedding; Introducing an adaptive weight matrix to dynamically adjust weights based on device usage; A=AdaptiveWeightMatrix(U) Among them, U indicates that the device uses an embedded vector set; AdaptiveWeightMatrix indicates a dynamically adjusted weight function; Extract and weight the features of single device data through Kalman filtering and multi-scale decomposition: Use Kalman filtering to estimate the state of device data and update the estimate based on usage; in, is the state estimate of the Kalman filter, which represents the state estimate after the observation data is given at time t; is the predicted state estimate of the Kalman filter, which represents the estimated state value at time t predicted based on the previous state and observation data at time t-1; K t is the Kalman gain matrix, used to update the state estimate; z t represents the observed value at time t; H t represents the observation model matrix, which is used to map variables in the state space to variables in the observation space; P t|t-1 represents the prediction error covariance matrix, which is used to describe the uncertainty of the state estimate at time t predicted at time t-1; Denotes the observation model matrix H t The transpose of R t represents the observation noise covariance matrix; Extract multi-scale features of device data using multi-scale decomposition; Among them, M(x) represents the multi-scale decomposition feature, which is used to extract the multi-scale features of the data; D j (x) represents the j-th level decomposition feature; J represents the number of scales in the multi-scale decomposition; j represents the scale index; x represents the input data, which is the updated data; Combine Kalman filtering and multi-scale decomposition to improve the attention mechanism; V2=xW2 Among them, d k2 =d u +d a represents the dimension normalization factor of the key matrix in the second data lake, d u Indicates the dimension of the embedding matrix used by the device, d a Represents the dimension of the adaptive weight matrix; IA S (x, U, A) represents the anomaly assessment result for the second data lake. A value of 1 indicates a serious anomaly, and a value of 0 indicates no anomaly.

6. The method for dynamic storage and correction based on meter reading data according to claim 5, characterized in that: The generator objective function is expressed as: Among them, min G Represents the loss function L that minimizes the generator G G , L G represents the loss function of the generator, E represents the expected value operation, z~p z (z) represents the noise distribution p z (z) represents the noise z sampled in, G(z) represents the candidate corrected data generated by the generator G for the noise z, and D(...) represents the score of the discriminator D for the data; Discriminator objective function: Among them, max D Represents the loss function L that maximizes the generator D D , L D represents the loss function of the discriminator, x~p data (x) represents the distribution of real data p data The real data x sampled in (x); Comprehensive confrontation formula: Among them, L GAN Represents the comprehensive adversarial loss function of the generative adversarial network; The specific steps for data correction are: For input data x, the probability D(x) output by the discriminator D is used to evaluate the authenticity of the data x; if D(x) is lower than the threshold τ, the data x needs to be corrected; for each data x that needs to be corrected, multiple candidate corrected data are generated by the generator G: {G(z i )∣z i ~N(0,1),i=1,2,…,N} Among them, N represents the number of data that need to be corrected, G(z i ) represents the effect of the generator G on the noise z i Generated candidate correction data, z i ~N(0,1) represents the noise z sampled from the standard normal distribution N(0,1) i , i represents the index; The generated candidate correction data are scored by the discriminator D and the attention mechanism results, and the data x with the highest score is selected. corrected As a result of the correction; Among them, argmax means taking the G(z that makes the expression reach the maximum value) i ); After the data correction is completed, the meter reading data is stored; when the data is displayed, the corrected data is displayed first, and the original data can be viewed through tracing.

7. A dynamic storage and correction system based on meter reading data using the method according to any one of claims 1 to 6, characterized in that: The collection unit obtains the original meter reading data by reading the meter; A transmission unit, which preprocesses and compresses the original meter reading data through blockchain, and stores the compressed data; The analysis unit integrates the compressed data into the data lake and uses the adversarial network improved by the attention mechanism to perform data analysis; The storage unit corrects and stores the meter reading data according to the analysis result.

8. A computer device comprising: Memory and processor; The memory stores a computer program, wherein the processor implements the steps of any one of the methods of claims 1-6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images