Power data compression and storage method and system based on edge internet-of-things proxy

Abstract

The invention discloses a power data compression and storage method and system based on an edge internet-of-things agent. The method comprises the steps of collecting power data, modeling the data and learning power features. And performing lossless compression and security encryption on the power data, and uploading the power data to a platform end. And receiving and analyzing the data at the platform end, reconstructing a data structure, and generating a final data table. According to the power data compression and storage method and system based on the edge internet of things agent, edge side intelligent research and judgment are achieved, the pressure of edge side data local storage and cloud side interaction data transmission network bandwidth is reduced, and the edge side minute-level data storage capacity and the data transmission efficiency are improved. Meanwhile, encryption measures in the data transmission process are involved, validity verification before and after data receiving is also involved, and the safety and integrity of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of power Internet of Things data processing, and specifically to a power data compression and storage method and system based on edge Internet of Things agent. Background Art

[0002] The amount of power data obtained by traditional compression storage methods is huge, which occupies a lot of hardware space and requires a large investment. However, the compressed sensing theory based on deep learning and data-driven can abandon the traditional "sampling first and then processing" method, and perform sampling and compression at the same time. The original signal is reconstructed through a small amount of data, which greatly reduces the hardware burden and greatly reduces the storage, processing and transmission costs of power data information. It is characterized by high compression ratio, fast transmission speed and large amount of data parallelism, which is conducive to the promotion and application of new technologies. In the process of signal reconstruction, this method can learn the correlation between sparse signals and realize signal recovery with minimal time and space cost. In addition, data parallel processing can accelerate the processing speed of auxiliary edge side and cloud devices, perform feature analysis on power data, thereby reducing data redundancy and transmission channel pressure and increasing data parallelism, and realizing ultra-real-time communication. In the actual process of assisting power companies to obtain real-time dynamic data, the data generated by smart grids have always been stored. In the past environment, it took about one cycle to analyze the data, which greatly consumed manpower and material resources. Real-time monitoring effectively solves this problem, making smart grid data processing fast and accurate. However, affected by many factors, this meticulous data also has many defects, which is very likely to cause network paralysis and server failure. In this way, it is impossible to guarantee speed and accuracy in time. Improving this situation is currently a huge challenge.

[0003] In addition, during the data transmission process, the channel side is often subject to adversarial attacks. If such attacks cannot be resisted, data leakage will also occur. Therefore, protecting the communication security on the channel side is as important as ensuring the security of power data. The hash check method uses the hash function before and after the channel data. By comparing the before and after values, it can be known whether the data content has been tampered with. However, how to effectively apply the hash check method to high-frequency power data to ensure the uniqueness of the hash function value is still a difficult problem to be solved. At the same time, fragile watermarking is a security guarantee in data circulation. During the data collection process, the relevant information of data collection is added to the data without affecting the use of the original data. The information hidden in the carrier ensures that the data is not tampered with. However, there are technical drawbacks in the application of existing fragile watermarking methods to high-frequency power data, such as large space occupied by hidden information and poor real-time transmission. Therefore, how to successfully apply the fragile watermarking method to high-frequency power data is a technical difficulty.

[0004] In these complex situations, we use deep learning networks to achieve modeling and feature learning of power data, and combine adaptive arithmetic coding and bit coding to complete lossless compression of power data, improve the compression efficiency of high-frequency power data, save resources during storage and transmission, and deploy power data compression storage micro-applications on edge IoT agents to solve the problems encountered in the storage and transmission of power data. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is that the existing fragile watermarking method has technical drawbacks when applied to high-frequency power data, such as the large space occupied by hidden information and poor real-time transmission. Therefore, how to successfully apply the fragile watermarking method to high-frequency power data is a technical difficulty.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a power data compression and storage method based on edge IoT agent, comprising: collecting power data, modeling the data, and learning power characteristics.

[0008] The power data is losslessly compressed and securely encrypted and uploaded to the platform.

[0009] Receive and parse data on the platform, reconstruct the data structure, and generate the final data table.

[0010] As a preferred solution of the power data compression and storage method based on edge IoT agent described in the present invention, wherein: the collecting power data, modeling the data, and learning power characteristics include collecting current, voltage and power data, and for power data with a length of n, modeling it as power data with a length of n, expressed as:

[0011] X={x 1 ,x 2 ,...,x n}

[0012] Among them, x i represents the hyperplane normal vector.

[0013] According to the bidirectional LSTM model to learn the power characteristics, the LSTM hidden unit is expressed as:

[0014] I t =σ(W i ·[h t-1 ,x t ]+b i ),

[0015] f t =σ(W f ·[ht-1 ,x t ]+b f ),

[0016] C t =f t ·C t-1 +I t tanh(W c ·[h t-1 ,x t ]+b c ),

[0017] O t =σ(W o ·[h t-1 ,x t ]+b o ),

[0018] h t =O t tanh(C t ).

[0019] Among them, t represents the time, W represents the weight matrix, I t represents the input gate, f t represents the forget gate, c t Indicates the cell state, o t represents the output gate, h t , indicating the hidden state.

[0020] As a preferred solution of the power data compression and storage method based on edge IoT agent described in the present invention, wherein: the learning of power characteristics according to the bidirectional LSTM model includes using one-hot to encode the character sequence into a K-dimensional vector sequence in the bidirectional LSTM model, using the K-dimensional vector sequence as the input of the network, and outputting the probability distribution of the next character.

[0021] The fully connected layer in the network is used to convert the hidden output into the output space of the power data dictionary length. The SoftMax function receives the output of the fully connected layer and outputs the probability distribution of the next predicted data. The predicted probability distribution is expressed as:

[0022] p(x i )=softmax(Wx i +b)

[0023] Among them, W represents the weight of this layer.

[0024] During the training process, loss is expressed as cross entropy:

[0025]

[0026] Among them, y(xij ) represents the true probability distribution, p(x ij ) represents the predicted probability distribution, and M represents the number of samples in the same batch.

[0027] As a preferred solution of the power data compression and storage method based on edge IoT agent described in the present invention, the lossless compression and secure encryption of power data includes that entropy reflects the compression performance of the compression algorithm for the cut segment X containing different N characters. Entropy is used as another performance metric for calculation, which is expressed as:

[0028]

[0029] Wherein, P(X) represents the calculated entropy value.

[0030] If the entropy before compression is greater than the entropy after compression, the compression algorithm is effective. If the output string is long and the number of repetitions is low, the greater the entropy after compression, the better the compression effect.

[0031] The compressed data is securely encrypted using a fragile watermark embedding algorithm.

[0032] As a preferred solution of the power data compression and storage method based on edge IoT agent described in the present invention, the uploading to the platform end includes securely encrypting the compressed data using a fragile watermark embedding algorithm and uploading it to the platform end through the MQTT protocol.

[0033] The watermark information is converted into an embeddable form through matrix coding technology and embedded in a table of power data in the carrier. Any slight change in the digital signal will cause the extracted watermark to be distorted. The degree of distortion of the extracted watermark can be used to determine whether the digital signal has been tampered with or destroyed.

[0034] Use the MQTT protocol to upload compressed power data to the platform. First, determine the structure of the data packet, including voltage, current, collection time, phase, and power. Design the MQTT topic structure.

[0035] Design clear topic naming rules for different data types and purposes. For voltage and current data with high real-time requirements, select QoS1, and for timestamps, select QoS0.

[0036] Authentication and authorization mechanisms are implemented during the upload process to ensure that only legitimate clients can publish and subscribe to topics.

[0037] As a preferred solution of the power data compression and storage method based on edge IoT agent described in the present invention, the receiving and parsing of data on the platform side includes reformatting the content of each row of the compressed table into the original power data message format, and using Hash verification to perform hash function on the data before and after the channel to ensure that the data channel has not been tampered with before and after, thereby ensuring the security of the channel and the model.

[0038] The platform first receives an external compressed package that contains the data to be processed, and restores the data to its original, uncompressed state using the Deflate decompression algorithm.

[0039] After decompression, the data is organized into key-value pairs, where the key is the column identifier and the value is the compressed encoded representation of the column data.

[0040] For each key, it is split into two parts by a predefined delimiter, including the level prefix and the data type.

[0041] Based on the identified data type, the platform selects a decoding algorithm to decode the value part. After decoding, each pair of level prefix: column array forms a part of the data. Each pair of level prefix: column array is combined to reconstruct the complete structure of the data.

[0042] As a preferred solution of the power data compression and storage method based on edge IoT agent described in the present invention, wherein: the generation of the final data table includes, when the data in all columns are decompressed and parsed, organizing the data into a table form, each row represents a record, and each column corresponds to a data attribute, which is convenient for data analysis, report generation and further data processing tasks.

[0043] An edge IoT agent-based power data compression storage system, characterized in that:

[0044] The acquisition and learning module collects power data, models the data, and learns power characteristics.

[0045] The encrypted upload module performs lossless compression and secure encryption on the power data and uploads it to the platform.

[0046] Accept the parsing module, receive and parse the data on the platform side, reconstruct the data structure, and generate the final data table.

[0047] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0048] A computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0049] The beneficial effects of the present invention are: realizing intelligent judgment on the edge side, reducing the pressure on the network bandwidth of local storage of edge data and cloud-edge interactive data transmission, and improving the minute-level data storage capacity and data transmission efficiency of the edge side. At the same time, it not only involves encryption measures for data during transmission, but also includes validity verification before and after data reception, ensuring the security and integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0051] Figure 1 An overall flow chart of a power data compression and storage method and system based on edge IoT agent provided for the first embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0053] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a power data compression and storage method based on edge IoT agent, comprising:

[0054] S1: Collect power data, model the data, and learn power characteristics.

[0055] It should be noted that, based on the technical perspective of deep learning, a lossless compression algorithm for power data is proposed to mine local information of data and improve computational efficiency. Adaptive arithmetic coding, the probability of model output is context-adaptive, the predicted probability is closer to the source probability, the total entropy value is smaller, and the value range in the final coding interval is wider, and the compression effect is better than static arithmetic coding. When using deep network training, the data is input into the model through iteration, and the prediction of the model is changed. We consider using the LSTM model to control the output of neurons through some gates to solve these problems. When training data, our environment is deployed on GeForce GTX 1080Ti. In order to apply the algorithm in real scenarios, we adjust the parameters of the model, find a suitable threshold between model accuracy and size, set the hidden layer features to 32, and finally extract 16 features in the fully connected layer to simplify the model, so that the algorithm can be deployed in the CPU environment and ensure the accuracy of the model. In real scenarios, the electricity collection meter is deployed in the user's living area, without too high configuration, so a lightweight model is needed. By controlling the parameters of the hidden layer to improve the model calculation speed, training takes about 2 hours on a single GPU. In real-world testing, we can achieve an average time of <0.1s for a single model call on the CPU. Unfortunately, as the length of the compressed data increases, the overall time consumption is still very long.

[0056] In the bidirectional LSTM model, we use one-hot to encode the character sequence into a K-dimensional vector sequence, take the K-dimensional vector sequence as the input of the network, output the probability distribution of the next character, minimize the cross entropy loss, and train the model weights using truncated backpropagation with a time step of 100. The state of the neural unit is retained across batches, and the Adam optimizer is used to solve the gradient explosion problem in Bi-LSTM. For power data of length n, the model is modeled as having a power data representation of length n:

[0057] X={x 1 ,x 2 ,...,x n}

[0058] Among them, x i represents the hyperplane normal vector.

[0059] According to the bidirectional LSTM model to learn the power characteristics, the LSTM hidden unit is expressed as:

[0060] I t =σ(W i ·[h t-1 ,x t ]+b i ),

[0061] f t =σ(Wf ·[h t-1 ,x t ]+b f ),

[0062] C t =f t ·C t-1 +I t tanh(W c ·[h t-1 ,x t ]+b c ),

[0063] O t =σ(W o ·[h t-1 ,x t ]+b o ),

[0064] h t =O t tanh(C t ).

[0065] Among them, t represents the time, W represents the weight matrix, I t represents the input gate, f t represents the forget gate, c t Indicates the cell state, o t represents the output gate, h t , indicating the hidden state.

[0066] It should be noted that the predicted probability of the model is obtained by iteratively inputting data into the model. The model can achieve the goal of reducing the size of the model while effectively predicting the probability distribution by adjusting the parameters of the model. The fully connected layer is used to convert the hidden output into the output space of the power data dictionary length. The SoftMax function receives the output of the fully connected layer and outputs the probability distribution of the next predicted data. The predicted probability distribution can be calculated by the following formula:

[0067] p(x i )=softmax(Wx i +b)

[0068] Among them, W represents the weight of this layer.

[0069] It should be noted that, in the case of power data, extracting bidirectional spatial features can better capture the dependencies between the previous and next steps, thereby increasing the accuracy of the prediction. During the training process, the cross entropy used by the loss is:

[0070]

[0071] Among them, y(x ij) represents the true probability distribution, p(x ij ) represents the predicted probability distribution, and M represents the number of samples in the same batch.

[0072] As shown in Table 1, the network layers and parameter sizes of the Bi-LSTM model can clearly show the operation method, and finally output a K-dimensional vector, which represents the probability distribution of the next character.

[0073] Table 1 BiLSTM model layer description

[0074] Layers Process OutputSize Layer1 Embedding [128,50,50] Layer2 Bi-LSTM [128,50,32] Layer3 Bi-LSTM [128,32] Layer4 FC [128,16] Layer5 FC [128,7820]

[0075] S2: Losslessly compress and securely encrypt the power data and upload it to the platform.

[0076] It should be noted that considering that the actual power consumption data of users is non-sinusoidal, the wavelet transform is invalid for comparing minute-level power data. Entropy, like compression ratio, is also an important parameter of the compression algorithm. It is the expected value of the information contained in each message. In some cases, it can reflect the compression performance of the algorithm for the cut segment X containing different N characters. We use entropy as another performance metric, calculated as:

[0077]

[0078] Wherein, P(X) represents the calculated entropy value.

[0079] It should be noted that when the entropy before compression is greater than the entropy after compression, it proves that the compression algorithm is effective. When the output string length is large and the number of repetitions is not high, the larger the entropy after compression, which means that the greater the difference between them, the better the compression effect. The results show that the entropy value of the algorithm after compression is lower than that of the classical algorithm. The analysis shows that the deep learning model has a positive impact on entropy, while the output probability of traditional coding is fixed and the entropy effect is not good. The prediction probability of the deep model after training will change adaptively, and the output is closer to the true value, which will save memory for subsequent encoding and compression, and the probability of adaptive change will improve CR and entropy.

[0080] The lossless compression algorithm is used to complete the lossless compression of current, voltage and power. This compression method can better protect the consumer rights and interests of users and ensure the accuracy of data.

[0081] The compressed data is securely encrypted using a fragile watermark embedding algorithm.

[0082] The compressed data is securely encrypted using a fragile watermark embedding algorithm and uploaded to the platform via the MQTT protocol.

[0083] The watermark information (specific identifier) ​​is converted into an embeddable form through matrix coding technology and embedded in a table of power data in the carrier. Any slight change in the digital signal will cause the extracted watermark to be distorted. The degree of distortion of the extracted watermark can be used to determine whether the digital signal has been tampered with or destroyed.

[0084] Use the MQTT protocol to upload compressed power data (including voltage, current, acquisition time, power, phase, etc.) to the platform. The following contents need to be specified:

[0085] First, determine the structure of the data packet, including:

[0086] Voltage: The unit is volt (V).

[0087] Current: The unit is ampere (A).

[0088] Collection time (Timestamp): Use UTC timestamp to unify time zone management.

[0089] Phase: L1, L2, L3, etc., used to distinguish the measurement values ​​of different circuits.

[0090] Power: It refers to the work done per unit time or the rate of energy conversion, and its unit is KW.

[0091] Design clear topic naming conventions for different data types and purposes:

[0092] sensor / area1 / phase{phase} / voltage: Publishes voltage data, where {phase} is replaced by the actual phase, such as L1.

[0093] sensor / area1 / phase{phase} / current: publishes current data.

[0094] sensor / area1 / timestamp: If all the data are collected at the same time, a timestamp topic can be published separately. Otherwise, each data packet contains its own timestamp.

[0095] For voltage and current data with high real-time requirements, QoS1 needs to be selected.

[0096] Timestamp updates may have lower QoS requirements, so select QoS 0.

[0097] Consider implementing authentication and authorization mechanisms during the process to ensure that only legitimate clients can publish or subscribe to specific topics.

[0098] S3: Receive and parse data on the platform, reconstruct the data structure, and generate the final data table.

[0099] The content of each row of the compressed table is reformatted into the original power data message format, and the hash function is used to check the data before and after the channel to ensure that the data channel has not been tampered with, thereby ensuring the security of the channel and the model.

[0100] The platform first receives an external compressed package that contains the data to be processed, and restores the data to its original, uncompressed state using the Deflate decompression algorithm.

[0101] After decompression, the data is organized into key-value pairs, where the key is the column identifier and the value is the compressed encoded representation of the column data.

[0102] For each key, it is split into two parts by a predefined delimiter, including the level prefix and the data type.

[0103] Based on the identified data type, the platform selects a decoding algorithm to decode the value part. After decoding, each pair of level prefix: column array forms a part of the data. Each pair of level prefix: column array is combined to reconstruct the complete structure of the data.

[0104] It should be noted that, taking the integer type in the data as an example:

[0105] ① Split column names:

[0106] Column name: user-data-age

[0107] After separation by the separator -, we get: user (level prefix), data (may be an intermediate layer, depending on the specific specifications, the explanation is omitted in this example to simplify the discussion), age (data type marker).

[0108] ②Analysis information:

[0109] The hierarchical prefix: user indicates that the data is part of the meter information and is used to locate the module or category to which the data belongs in the big data architecture.

[0110] Data type: Inferred from age, this column of data should be voltage information, and the expected data type is integer (int).

[0111] ③Decoding preparation:

[0112] After knowing that the data type is integer, the system will compress the integer array in gzip format, perform gzip decompression, and then correctly decode it into the actual voltage value according to the storage format of the integer data (such as complement code, original code, etc.).

[0113] ④Application decoding:

[0114] The decompressed code block is a byte stream encoded in integer format, which the system converts into an intuitive sequence of integers, such as [25,30,35,...], which represents the voltage of each meter.

[0115] When the data in all columns are decompressed and parsed, the data is organized into a table format, with each row representing a record and each column corresponding to a data attribute, which facilitates data analysis, report generation, and further data processing tasks.

[0116] The above embodiments also include a power data compression storage system based on edge IoT agent, specifically:

[0117] The acquisition and learning module collects power data, models the data, and learns power characteristics.

[0118] The encrypted upload module performs lossless compression and secure encryption on the power data and uploads it to the platform.

[0119] Accept the parsing module, receive and parse the data on the platform side, reconstruct the data structure, and generate the final data table.

[0120] The computer device may be a server. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data cluster data of a power monitoring system. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a power data compression and storage method based on an edge IoT agent is implemented.

[0121] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0122] Example 2 is an embodiment of the present invention, which provides a power data compression and storage method and system based on edge IoT agent. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0123] The experimental object is the compression efficiency and data security of power data. The experiment uses the existing data processing method based on traditional compression algorithm and the compression storage method of the present invention for comparison.

[0124] The data acquisition device collects power data such as voltage, current, and power. The edge computing device is used to implement data modeling, compression, and encryption processing. The platform receives the compressed and encrypted data, parses, and reconstructs the data table. The MQTT protocol is used to transmit data.

[0125] Prior art steps:

[0126] Collect power data and transmit it directly to the platform.

[0127] Data compression is done using the traditional LZ compression algorithm, which reduces the amount of data by finding repetitive patterns in the data.

[0128] The compressed data is encrypted using the AES encryption algorithm and then uploaded to the platform.

[0129] The platform receives, decompresses and decrypts the data and reconstructs the data table.

[0130] Steps of the present invention:

[0131] Collect power data such as current, voltage, and power, and perform bidirectional LSTM modeling on the data on the edge computing device to learn the time series characteristics of the data.

[0132] According to the output of the LSTM model, the data is losslessly compressed, and entropy is used as a compression performance metric to ensure the effectiveness of the compression algorithm.

[0133] The compressed data is securely encrypted using a fragile watermark embedding algorithm, and the encrypted data is uploaded to the platform using the MQTT protocol.

[0134] The platform receives and parses the data packet, uses the Deflate algorithm to decompress the data, and ensures data integrity through hash verification.

[0135] Finally, the data is reconstructed into a tabular form to facilitate subsequent data analysis and processing.

[0136] During the experiment, power data in the same time period was collected, and the volume and encryption performance of the data before and after compression were recorded. At the same time, a bidirectional LSTM model was used to extract features from the data, optimize the compression efficiency, and ensure the security of data transmission through a fragile watermark embedding algorithm. The experimental results are shown in Table 2.

[0137] Table 2 Experimental data comparison table

[0138]

[0139]

[0140] It can be seen from the table data that the compression storage method based on the present invention is significantly superior to the prior art in terms of data compression efficiency, error rate and security.

[0141] The compression rate of traditional compression algorithms is generally low, and the size of compressed data is about 50-60% of the original data. However, the present invention uses the LSTM model to learn the time series characteristics of power data and combines it with the entropy optimization algorithm to achieve an average compression rate of more than 70%. This shows that the present invention can significantly reduce the demand for data storage and has significant advantages in large-scale power data processing.

[0142] After data decompression, the error rate of traditional algorithms is relatively high, while the lossless compression algorithm of the present invention controls the error rate to between 0.01% and 0.03%, maintaining high-precision data recovery. This is particularly important for application scenarios that require accurate recovery of power data, such as power grid dispatching and real-time monitoring.

[0143] The present invention improves the security of data transmission by embedding fragile watermarks, and combines the identity authentication mechanism of the MQTT protocol to ensure that only authorized devices can upload and parse data. The average security evaluation score in the experiment was 9.5 points, which is significantly higher than the performance of traditional encryption algorithms, reflecting the advantages of the present invention in data transmission and storage security.

[0144] Through the above comparative analysis, it can be clearly seen that the present invention has significant innovations and advantages in data compression efficiency, data recovery accuracy and data security, is suitable for large-scale power data processing and storage application scenarios, and improves the overall performance and reliability of the system.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A power data compression and storage method based on edge IoT agent, characterized in that: include: Collect power data, model the data, and learn power characteristics; Losslessly compress and securely encrypt power data and upload it to the platform; Receive and parse data on the platform, reconstruct the data structure, and generate the final data table.

2. The power data compression and storage method based on edge IoT agent according to claim 1, characterized in that: The collecting of power data, modeling of the data, and learning of power features include collecting current, voltage and power data, and for power data of length n, modeling as power data of length n, expressed as: X={x1,x2,...,x n } Among them, x i represents the hyperplane normal vector; According to the bidirectional LSTM model to learn the power characteristics, the LSTM hidden unit is expressed as: I t =σ(W i ·[h t-1 ,x t ]+b i ), f t =σ(W f ·[h t-1 ,x t ]+b f ), C t =f t ·C t-1 +I t ·tanh(W c ·[h t-1 ,x t ]+b c ), The t =σ(W o ·[h t-1 ,x t ]+b o ), h t =O t ·tanh(C t ). Among them, t represents the time, W represents the weight matrix, I t represents the input gate, f t represents the forget gate, c t Indicates the cell state, o t represents the output gate, h t , indicating the hidden state.

3. The power data compression and storage method based on edge IoT agent according to claim 2, characterized in that: The learning of power features according to the bidirectional LSTM model includes encoding the character sequence into a K-dimensional vector sequence using one-hot in the bidirectional LSTM model, using the K-dimensional vector sequence as an input of the network, and outputting a probability distribution of the next character; The fully connected layer in the network is used to convert the hidden output into the output space of the power data dictionary length. The SoftMax function receives the output of the fully connected layer and outputs the probability distribution of the next predicted data. The predicted probability distribution is expressed as: p(x i )=softmax(Wx i +b) Among them, W represents the weight of this layer; During the training process, loss is expressed as cross entropy: Among them, y(x ij ) represents the true probability distribution, p(x ij ) represents the predicted probability distribution, and M represents the number of samples in the same batch.

4. The power data compression and storage method based on edge IoT agent according to claim 3 is characterized in that: The lossless compression and secure encryption of the power data includes that entropy reflects the compression performance of the compression algorithm for the cut segment X containing different N characters; entropy is used as another performance metric for calculation, which is expressed as: Where P(X) represents the calculated entropy value; If the entropy before compression is greater than the entropy after compression, the compression algorithm is effective; if the output string is long and the number of repetitions is low, the greater the entropy after compression, the better the compression effect; The compressed data is securely encrypted using a fragile watermark embedding algorithm.

5. The power data compression and storage method based on edge IoT agent according to claim 4, characterized in that: The uploading to the platform end includes securely encrypting the compressed data using a fragile watermark embedding algorithm and uploading it to the platform end via the MQTT protocol; The watermark information is converted into an embeddable form through matrix coding technology, and embedded in the table of power data in the carrier. Any slight change in the digital signal will cause the extracted watermark to be distorted. According to the degree of distortion of the extracted watermark, it can be determined whether the digital signal has been tampered with or destroyed. Use the MQTT protocol to upload compressed power data to the platform. First, determine the structure of the data packet, including voltage, current, collection time, phase, and power; design the MQTT topic structure; Design clear topic naming rules for different data types and purposes. For voltage and current data with high real-time requirements, select QoS1, and for timestamps, select QoS0; Authentication and authorization mechanisms are implemented during the upload process to ensure that only legitimate clients can publish and subscribe to topics.

6. The power data compression and storage method based on edge IoT agent according to claim 5, characterized in that: The receiving and parsing of data on the platform side includes reformatting each row of the compressed table into the original power data message format, and using a hash check to perform a hash function on the data before and after the channel to ensure that the data channel has not been tampered with before and after, thereby ensuring the security of the channel and the model; The platform first receives an external compressed package that contains the data to be processed. It then uses the Deflate decompression algorithm to restore the data to its original, uncompressed state. After decompression, the data is organized into key-value pairs, where the key is the column identifier and the value is the compressed encoding representation of the column data; For each key, split it into two parts, including the level prefix and the data type, by a predefined separator; Based on the identified data type, the platform selects a decoding algorithm to decode the value part; After decoding, each pair of level prefix:column array forms a part of the data; Combine each pair of level prefix:column arrays to reconstruct the complete structure of the data.

7. The power data compression and storage method based on edge IoT agent according to claim 6, characterized in that: The generating of the final data table includes, when the data of all columns are decompressed and parsed, organizing the data into a table form, each row represents a record, and each column corresponds to a data attribute, which is convenient for data analysis, report generation and further data processing tasks.

8. A power data compression storage system based on edge IoT agent using the method according to any one of claims 1 to 7, characterized in that: The acquisition and learning module collects power data, models the data, and learns power characteristics; The encryption upload module performs lossless compression and secure encryption on the power data and uploads it to the platform. Accept the parsing module, receive and parse the data on the platform side, reconstruct the data structure, and generate the final data table.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images