Edge-computing-based transformer area intelligent electric meter data compression transmission method and system

Abstract

This invention relates to the field of electricity meter data transmission technology, and particularly to a method and system for compressing and transmitting data from smart meters in a distribution area based on edge computing. The method includes: deploying an edge computing gateway on the transformer side of the distribution area to periodically receive raw electrical energy data sequences uploaded by all smart meters within the area. The gateway employs improved run-length encoding, dynamically adjusting segmentation thresholds based on the current electricity consumption pattern to complete initial compression. Then, it combines adaptive dictionary encoding to dynamically construct a local dictionary based on the statistical characteristics of data blocks for secondary compression, generating high-compression-ratio data packets. The gateway locally maintains a meter behavior feature database, screening and marking abnormal data packets. Abnormal data packets are transmitted in real-time without loss, while normal data packets are periodically batch-aggregated and transmitted. This invention can reduce transmission bandwidth usage, optimize transmission resource allocation, and shorten data processing and response latency.

Description

Technical Field

[0001] This invention relates to the field of electricity meter data transmission technology, and in particular to a method and system for data compression and transmission of smart meters in distribution areas based on edge computing. Background Technology

[0002] During the data transmission process of smart meters in the distribution area, existing technologies mostly use concentrators to receive the raw electrical energy data uploaded by smart meters in the distribution area, and directly compress the raw data using run-length encoding with a fixed threshold or a single dictionary encoding. The compression algorithm is not optimized by combining electricity consumption patterns and data statistical characteristics. In addition, the data transmission mostly adopts a unified transmission mechanism, without screening for data anomalies, and without deploying dedicated processing equipment at the edge.

[0003] Fixed-threshold run-length encoding cannot adapt to data changes under different power consumption patterns, and a single dictionary encoding is difficult to dynamically adjust based on data statistical characteristics, resulting in low data compression ratios and a large amount of redundant data consuming transmission bandwidth. The unified transmission mechanism does not distinguish between data anomalies; normal and abnormal data are transmitted using the same method, which affects the timeliness of abnormal data transmission and wastes resources for normal data transmission. Furthermore, the lack of local processing at the edge results in high data transmission latency and an inability to respond quickly to data anomalies.

[0004] An edge computing gateway needs to be deployed on the transformer side of the distribution area to process local data, optimize the compression algorithm to improve the data compression ratio, dynamically adjust the compression parameters based on power consumption patterns and data statistical characteristics, and perform anomaly screening on the compressed data at the edge side. A differentiated transmission mechanism is adopted for abnormal and normal data to make up for the shortcomings of existing technologies in compression efficiency, transmission timeliness and resource utilization. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a data compression and transmission method and system for smart meters in distribution areas based on edge computing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a data compression and transmission method for smart meters in distribution areas based on edge computing, comprising a data compression and transmission method running on an edge computing gateway, and a data restoration method running on a main station system, wherein the data compression and transmission method includes: An edge computing gateway is deployed on the transformer side of each distribution area to periodically receive the raw electrical energy data sequence uploaded by all smart meters under the distribution area; Within the edge computing gateway, the received raw electrical energy data sequence is initially compressed using an improved run-length encoding algorithm. The improved run-length encoding algorithm dynamically adjusts its segmentation threshold based on the current electricity consumption pattern to generate initially compressed data blocks and corresponding segmentation metadata. Based on the segmented metadata, the initially compressed data block is compressed a second time using an adaptive dictionary encoding algorithm. The adaptive dictionary encoding algorithm dynamically constructs a local dictionary based on the statistical characteristics of the data block to generate a high compression ratio data packet. The edge computing gateway maintains a meter behavior feature library locally, which stores the historical electricity consumption features of each smart meter. The meter behavior feature library is used to screen and mark outliers in the high compression ratio data packets. For high-compression ratio data packets marked as abnormal, a real-time lossless transmission channel is initiated; for high-compression ratio data packets not marked as abnormal, a periodic batch aggregation transmission mechanism is adopted. The data restoration method includes: After receiving the transmission frame from the edge computing gateway, the main station system performs reverse decoding according to the agreed compression encoding rules to recover the original time-series electrical energy data. The main station system will compare the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway to complete the data consistency verification.

[0007] As a further aspect of the present invention, the improved run-length encoding algorithm dynamically adjusts its segmentation threshold based on the current electricity consumption pattern, including: Statistical feature analysis is performed on the original electrical energy data sequence within the current time window to extract its mean, variance, and distribution characteristics of the difference between adjacent data points; Based on the distribution characteristics of the mean, variance, and difference between adjacent data points, a preset pattern matching table is queried to determine the current electricity consumption mode, which includes a stable mode, a fluctuating mode, a peak mode, and a valley mode. A dynamic segmented threshold reference coefficient is associated with each of the aforementioned electricity consumption modes, wherein the segmented threshold reference coefficients corresponding to the fluctuation mode and the peak mode are less than the segmented threshold reference coefficients corresponding to the steady mode and the valley mode. The product of the segmentation threshold benchmark coefficient and the overall range of the original electrical energy data sequence is used as the real-time dynamic segmentation determination threshold for the current time window. During run-length encoding, when the difference in the numerical change of the continuously read raw electrical energy data is less than the real-time dynamic segmentation determination threshold, they are merged into a run-length segment and recorded with representative value and length information to form the preliminary compressed data block and the segmentation metadata that records the segmentation boundary information.

[0008] As a further aspect of the present invention, based on the segmented metadata, an adaptive dictionary encoding algorithm is used to perform secondary compression on the initially compressed data block, including: The segmented metadata is analyzed to identify the length distribution characteristics and the range of representative values ​​for each run segment in the preliminary compressed data block. Based on the length distribution characteristics and the range of representative values, a local dictionary suitable for the current preliminary compressed data block is dynamically constructed in memory. The local dictionary contains frequently occurring run-length codes and representative value codes. Traverse the initial compressed data block and match the length and representative value of each run segment with the codewords in the local dictionary; The high-frequency run lengths and representative values ​​that are successfully matched are replaced with the corresponding codewords with shorter code lengths in the local dictionary. For low-frequency run lengths and representative values ​​that do not appear in the local dictionary, their original codes are retained but a specific identifier is added. The encoded codeword sequence, the local dictionary, and the decoding rule index are packaged to generate the high-compression data packet.

[0009] As a further aspect of the present invention, the high compression ratio data packets are screened for and marked using the meter behavior feature database, including: The original electrical energy data sequence at the current moment is decoded from the high compression ratio data packet; The reference electricity consumption curve of the corresponding smart meter is retrieved from the meter behavior feature library. The reference electricity consumption curve is obtained by statistical learning from historical normal electricity consumption data of the same period. Calculate the dynamic time warping distance between the raw electrical energy data sequence at the current moment and the reference electricity consumption curve at the corresponding moment; The dynamic time-normalized distance is compared with a preset distance threshold; If the dynamic time warping distance exceeds the distance threshold, the current data packet is determined to contain potentially abnormal power consumption data, and an abnormal flag is added to the high compression ratio data packet; otherwise, it is considered a normal data packet. For high compression ratio data packets marked as abnormal, key abnormality summary information is extracted from the original electrical energy data sequence, including the time of abnormality occurrence, duration of abnormality, and magnitude of abnormality.

[0010] As a further aspect of the present invention, a real-time lossless transmission channel is initiated for the high-compression ratio data packets marked as abnormal, while a periodic batch aggregation transmission mechanism is employed for the high-compression ratio data packets not marked as abnormal, including: A dual-queue cache is established inside the edge computing gateway, namely a real-time transmission queue and a batch transmission queue; The high-compression ratio data packets identified as anomalous and their corresponding anomalous summary information are immediately placed into the real-time transmission queue; The high compression ratio data packets identified as unmarked anomalies are categorized and cached in the batch transmission queue according to the identifier and timestamp of their corresponding smart meters. The real-time transmission queue adopts an event-triggered mechanism. Once the queue is not empty, it initiates the instant transmission of a single packet to the main station system. The batch transmission queue adopts a time-triggered mechanism. When the number of accumulated unmarked abnormally high compression ratio data packets reaches a preset threshold or a fixed aggregation sending period is reached, subsequent batch aggregation transmission is triggered.

[0011] As a further aspect of the present invention, the method further includes: In periodic batch aggregation transmission, high compression ratio data packets with unmarked anomalies within multiple time windows are merged into aggregated data blocks according to preset aggregation rules; Lightweight error control coding is used to encapsulate and protect the aggregated data block, generating a transmission frame with verification. On the communication link between the edge computing gateway and the main station system, differentiated network service quality policies are allocated to the real-time lossless transmission channel and batch transmission frames to prioritize the bandwidth and low latency of the real-time lossless transmission channel. After receiving the transmission frame, the master station system performs reverse decoding according to the compression encoding rules agreed upon with the edge computing gateway to recover the original time-series electrical energy data; The main station system will compare the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway during anomaly screening to complete the data consistency verification. High-compression data packets with unmarked anomalies from multiple time windows are merged into aggregated data blocks according to preset aggregation rules, specifically including: A predetermined number of high-compression data packets with unmarked anomalies are read from the batch transmission queue in chronological order. Parse the header information of each high-compression data packet to extract its corresponding smart meter identifier, data start time, and data end time; All high-compression data packets are grouped according to the smart meter identifier; Within the same smart meter identifier group, the payload data portions of multiple high-compression ratio data packets are sequentially spliced ​​together according to the order of their data start times. A new aggregation header is generated for the spliced ​​overall data block. The aggregation header contains all smart meter identifiers of the aggregated data, the total time range of the aggregated data, and the offset index of each high-compression data packet within the aggregated data block. The aggregated header is combined with the concatenated overall data block to form the aggregated data block.

[0012] As a further aspect of the present invention, lightweight error control coding is used to encapsulate and protect the aggregated data block, including: Cyclic redundancy check calculation is performed on the aggregated data block to generate a first-level check code; The aggregated data block is divided into multiple data fragments of a fixed size; Calculate the checksum for each data fragment and generate second-level fragment-level checksum information; The first-level checksum, all data fragments and their corresponding second-level fragment-level checksums are assembled according to a specific frame format, which includes a frame start flag, a length field, an address field, a data type identifier, an aggregated data block, a first-level checksum, second-level fragment-level checksums and a frame end flag. The assembled frame structure is the transmission frame with verification.

[0013] As a further aspect of the present invention, differentiated network service quality strategies are assigned to the real-time lossless transmission channel and the batch transmission frames, including: In the network protocol stack of the edge computing gateway, two logical data transmission channels with different priorities are configured; Data packets flowing to the real-time lossless transmission channel are marked as high priority, and a high-priority quality of service policy is configured. The high-priority quality of service policy includes guaranteed minimum bandwidth, lower queue scheduling latency, and lower packet loss weight. The data stream containing the batch transmission frames is marked as low priority, and a low-priority quality of service policy is configured, which includes best-effort bandwidth allocation, allowing large scheduling delays, and prioritizing packet drop during congestion. During data transmission, data packets are routed to the corresponding logical data transmission channel for processing and transmission based on their type.

[0014] As a further aspect of the present invention, after receiving the transmission frame, the master station system performs reverse decoding according to the compression encoding rules agreed upon with the edge computing gateway, including: After receiving the transmission frame, the master station system first parses its frame format and identifies the data type identifier; If the data type identifier indicates real-time lossless transmission data, then the high compression ratio data packet is directly extracted and decoded according to the agreed dictionary encoding rules and dynamic run-length encoding rules to obtain the original electrical energy data sequence and anomaly summary information; If the data type identifier indicates that the data is being transmitted in batches, then the aggregated data blocks are extracted, the aggregated header is parsed, and each independent high-compression data packet is separated according to the offset index. For each independent high-compression data packet, it is also decoded according to the agreed dictionary encoding rules and dynamic run-length encoding rules to recover the original energy data sequence of each smart meter in different time windows; The main station system compares the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway during anomaly screening to complete data consistency verification, specifically including: For data received and decoded through a real-time lossless transmission channel, the master station system performs logical verification on the decoded raw electrical energy data sequence and the accompanying anomaly summary information to confirm that the features described in the anomaly summary information do indeed exist in the data sequence. For normal data received and decoded through batch aggregation transmission, the main station system seamlessly splices it with abnormal data received from the real-time channel from adjacent time periods of the same electricity meter in the time series, and checks the continuity and rationality of the data at the splicing point. The main station system calculates the global statistical characteristics of all recovered data and compares them with the statistical data summaries periodically uploaded by the edge computing gateway to verify whether the data has been distorted or lost during transmission and decompression. If the verification finds an inconsistency, a data retransmission request is initiated to the corresponding edge computing gateway, and the data is repaired using a backup redundancy mechanism.

[0015] As a further aspect of the present invention, the present invention also includes a data compression and transmission system for smart meters in distribution areas based on edge computing. The system includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the data compression and transmission method for smart meters in distribution areas based on edge computing as described above.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: Within the edge computing gateway, an improved run-length encoding algorithm is applied to initially compress the received raw electrical energy data sequence. This improved run-length encoding algorithm dynamically adjusts its segmentation threshold based on the current power consumption pattern, generating initially compressed data blocks and corresponding segmentation metadata. Then, based on the segmentation metadata, an adaptive dictionary encoding algorithm is used to perform secondary compression on the initially compressed data blocks. This adaptive dictionary encoding algorithm dynamically constructs a local dictionary based on the statistical characteristics of the data blocks, generating high-compression-ratio data packets. Compared to conventional fixed-threshold run-length encoding and single-dictionary encoding, dynamically adjusting the segmentation threshold can adapt to data fluctuations under different power consumption patterns, dynamically constructing a local dictionary can accurately match data statistical characteristics, and the secondary compression method effectively reduces data redundancy. The generated high-compression-ratio data packets reduce transmission bandwidth consumption, optimizing both the adaptability and efficiency of data compression. Local compression at the edge also shortens data processing latency.

[0017] A meter behavior feature database is maintained locally on the edge computing gateway. This database stores the historical electricity consumption characteristics of each smart meter. This database is used to screen and mark outliers in high-compression ratio data packets. Real-time lossless transmission is initiated for high-compression ratio data packets marked as anomalies, while a periodic batch aggregation transmission mechanism is used for unmarked high-compression ratio data packets. The meter behavior feature database can accurately capture the electricity consumption patterns of each meter, enabling precise screening and marking of outliers in data packets. The differentiated transmission mechanism allows for rapid transmission of abnormal data, ensuring timely response to anomalies. Batch aggregation transmission of normal data reduces the number of transmissions and lowers transmission resource consumption. Local anomaly screening on the edge side eliminates the need to upload all data to the cloud for processing, further reducing data transmission and response latency, and making the allocation of transmission resources more rational.

[0018] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0019] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart of the edge computing-based smart meter data compression and transmission method for the present invention. Figure 2 A flowchart illustrating the dynamic adjustment of the segmentation threshold for the improved run-length encoding algorithm; Figure 3 This is a flowchart of a dual-queue differential transmission mechanism. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] See Figure 1 The core implementation of this invention is to deploy an edge computing gateway with computing and storage capabilities on the transformer side of each distribution area where data aggregation is required. This edge computing gateway periodically receives the raw electrical energy data sequence uploaded by all smart meters under the distribution area from the communication network it manages.

[0023] Inside the edge computing gateway, an improved run-length encoding algorithm is applied to the aggregated raw power data sequence to perform preliminary compression processing. This improved run-length encoding algorithm can dynamically adjust its segmentation judgment threshold according to the analyzed current power consumption pattern. This process generates preliminary compressed data blocks and corresponding segmentation metadata used to describe the segmentation status.

[0024] Based on the generated segmented metadata, an adaptive dictionary encoding algorithm is used to perform secondary compression on the initially compressed data block. The adaptive dictionary encoding algorithm dynamically constructs a local dictionary based on the statistical characteristics of the current data block, and encodes the data into a high compression ratio data packet through this local dictionary.

[0025] The edge computing gateway maintains a continuously updated meter behavior feature library locally. This library stores the historical electricity consumption characteristics of each smart meter. Using this meter behavior feature library, outlier screening and marking operations can be performed on the generated high-compression data packets.

[0026] For high-compression data packets that are marked as potentially abnormal after screening, the system will initiate a real-time lossless transmission channel for priority uploading; while for normal high-compression data packets that are not marked as abnormal, a periodic batch aggregation transmission mechanism will be used for delayed aggregation and transmission.

[0027] In one embodiment of the present invention, the implementation process of the improved run-length encoding and adaptive dictionary encoding is described, see reference. Figure 2The system performs statistical feature analysis on the raw electrical energy data sequence within the current time window, extracting statistical features including the distribution characteristics of the sequence's mean, variance, and differences between adjacent data points. Based on the obtained distribution characteristics of the mean, variance, and differences between adjacent data points, the edge computing-based smart meter data compression and transmission system queries a preset pattern matching table to determine the current electricity consumption pattern. The defined electricity consumption patterns include stable mode, fluctuating mode, peak mode, and valley mode. A dynamic segmentation threshold reference coefficient is associated with each preset electricity consumption pattern. The segmentation threshold reference coefficients associated with fluctuating mode and peak mode are smaller, while those associated with stable mode and valley mode are larger. The product of the determined segmentation threshold reference coefficient and the overall range of the currently processed raw electrical energy data sequence is used as the real-time dynamic segmentation judgment threshold for the current time window. During the run-length encoding process, when the difference in the numerical change of the continuously read raw electrical energy data is less than the calculated real-time dynamic segmentation judgment threshold, this continuous data is merged and recorded as a run segment. A representative value and the length information of the segment are used to characterize the continuous data segment. The set of these representative values ​​and length information constitutes the initial compressed data block, and segment metadata is generated to record the boundary information of each segment.

[0028] Based on the obtained segmented metadata, an adaptive dictionary encoding algorithm is used to perform secondary compression on the initial compressed data block. The segmented metadata is parsed to identify the length distribution characteristics (i.e., the characteristics obtained by statistically analyzing the length information of each segment) and the range of representative values ​​for each run segment in the initial compressed data block. Based on the identified length distribution characteristics and the range of representative values, a local dictionary suitable for the current initial compressed data block is dynamically constructed in the memory of the edge computing gateway. This local dictionary contains the run length codes and representative value codes that appear frequently in the current data block. The system traverses the entire initial compressed data block, matching the length and representative value of each run segment with the codewords in the local dictionary. Successfully matched high-frequency run lengths and representative values ​​are replaced with corresponding codewords in the local dictionary that have shorter code lengths. For low-frequency run lengths and representative values ​​that do not appear in the local dictionary, their original codes are retained but a specific identifier is added for differentiation. The encoded codeword sequence, the used local dictionary, and the decoding rule index are packaged together to generate the final high-compression ratio data packet.

[0029] In practical implementation, consider a residential area with an edge computing gateway deployed, connected to 1000 smart meters. The edge computing gateway collects electricity data every 15 minutes. The raw electricity data sequence consists of the current active power reading uploaded by each meter, in kilowatts. Taking the data of a specific meter during a typical morning from 8:00 to 12:00 as an example, the raw electricity data sequence received by the edge computing gateway is [0.52,0.51,0.53,0.80,5.20,5.21,5.19,1.50,1.51,1.49,0.50,0.52,0.51,0.50,0.50,0.51,0.50].

[0030] The edge computing gateway performs statistical feature analysis on the raw power data sequence within the current time window. First, it extracts the mean of the sequence, i.e., calculates the arithmetic mean of all data points, yielding a mean of approximately 1.64 kW. Next, it extracts the variance of the sequence, i.e., calculates the average of the squares of the differences between each data point and the mean, used to measure the degree of data fluctuation, yielding a variance of approximately 2.89. Then, it extracts the distribution characteristics of the differences between adjacent data points, specifically by calculating the absolute difference between the next data point and the previous data point in the sequence, thus obtaining the difference sequence [0.01, 0.02, 0.27, 4.40, 0.01, 0.02, 3.69, 0.01, 0.02, 0.99, 0.02, 0.01, 0.01, 0.00, 0.01, 0.01]. Analyzing this difference sequence reveals not only significantly large differences such as 4.40 and 3.69, but also statistical quantification. For example, 12 data points in this sequence have differences less than 0.1, accounting for 75% of the total 16 differences, indicating that most differences are at a low level. Based on the analysis of the mean of 1.64 kW, variance of 2.89, and the distribution characteristics of significantly large differences and a high proportion of small differences in the difference sequence, the edge computing gateway queries a preset pattern matching table to determine that the current power consumption pattern is a peak mode.

[0031] In practical implementation, the edge computing gateway performs statistical feature analysis on the raw power data sequence within the current time window, extracts the mean of the raw power data sequence (calculated as 1.64 kW), extracts the variance of the raw power data sequence (calculated as 2.89), and extracts the distribution characteristics of the difference between adjacent data points. Specifically, it calculates the absolute difference between the next data point and the previous data point in the sequence, resulting in a difference sequence [0.01, 0.02, 0.27, 4.40, 0.01, 0.02, 3.69, 0.01, 0.02, 0.99, 0.02, 0.01, 0.01, 0.00, 0.01, 0.01]. Analyzing the characteristics of the difference sequence, most differences are less than 0.1, but there are significantly large differences such as 4.40 and 3.69. Based on the extracted mean, variance, and differences between adjacent data points, the edge computing gateway queries a pre-defined pattern matching table. This table defines power consumption patterns corresponding to different statistical characteristic ranges. For example, a mean below 1 kW and a variance less than 0.1 correspond to a valley pattern, while a mean between 1 and 2 kW but a variance greater than 2 and a large difference exceeding 3.0 corresponds to a peak pattern. Since the current sequence has a mean of 1.64 kW within the 1-2 kW range, a variance of 2.89 greater than 2, and differences exceeding 3.0 such as 4.40 and 3.69 in the difference sequence, the pre-defined pattern matching table determines that the current power consumption pattern is a peak pattern.

[0032] The preset pattern matching table associates a dynamic segmented threshold base coefficient with each electricity consumption pattern. The segmented threshold base coefficients corresponding to fluctuating and peak patterns are smaller than those corresponding to stable and valley patterns. In this example, the segmented threshold base coefficient associated with the peak pattern is 0.02, the segmented threshold base coefficient associated with the stable pattern is 0.05, and the segmented threshold base coefficient associated with the valley pattern is 0.08. The product of the determined segmented threshold base coefficient and the overall range of the original energy data sequence is used as the real-time dynamic segmentation judgment threshold for the current time window. The overall range of the original energy data sequence is the difference between the maximum value of 5.21 kW and the minimum value of 0.50 kW, which is 4.71 kW. The calculated real-time dynamic segmentation judgment threshold is 0.02 multiplied by 4.71, which is approximately equal to 0.0942 kW.

[0033] During run-length encoding, when the difference in the numerical changes of continuously read raw electrical energy data is less than the real-time dynamic segmentation threshold, they are merged into a single run segment. Traversing the example raw electrical energy data sequence, starting from the first data point 0.52, the difference between 0.52 and 0.51 (0.01) is less than 0.0942, and the difference between 0.51 and 0.53 (0.02) is less than 0.0942. However, the difference between 0.53 and the next data point 0.80 (0.27) is greater than 0.0942. Therefore, the first three data points [0.52, 0.51, 0.53] form the first run segment, recorded with a representative value of 0.52 and a length of 3. Similarly, in the subsequent data, [0.80] forms a segment, [5.20, 5.21, 5.19] forms a segment, [1.50, 1.51, 1.49] forms a segment, and [0.50, 0.52, 0.51, 0.50, 0.50, 0.51, 0.50] forms a segment. This generates a preliminary compressed data block, containing the representative value and length information of each segment, and generates segment metadata that records the segment boundary information. The segment metadata records the start and end indices of each segment in the original sequence.

[0034] In some embodiments, a secondary compression is performed on the initially compressed data block using an adaptive dictionary encoding algorithm based on segment metadata. The segment metadata is parsed to identify the length distribution characteristics and representative value range of each run segment in the initially compressed data block. In the example, the segment lengths are 1 and 3, and the representative values ​​are mainly distributed between 0.50 and 5.21. Based on the length distribution characteristics and representative value range, a local dictionary suitable for the current initially compressed data block is dynamically constructed in memory. The initially compressed data block is traversed, and the frequency of occurrence of run lengths and representative values ​​is counted. In the example, length 3 occurs 4 times, length 1 occurs once, and representative values ​​0.52, 5.20, 1.50, and 0.50 each occur once. The constructed local dictionary contains high-frequency run length codes and representative value codes; for example, the short codeword "00" is assigned to the high-frequency length 3, and the short codeword "010" is assigned to the representative value 0.50. The initially compressed data block is traversed, and the length and representative value of each run segment are matched with the codewords in the local dictionary. The successfully matched high-frequency run-lengths and representative values ​​are replaced with their shorter codewords from the local dictionary. For example, a segment of length 3 is replaced with "00", and the representative value 0.50 is replaced with "010". Low-frequency run-lengths and representative values ​​not found in the local dictionary are retained in their original encoding but with a specific identifier added. For example, the representative value 0.80 is not in the dictionary, so its binary encoding is retained but prefixed with "111". The encoded codeword sequence, the local dictionary, and the decoding rule index are packaged to generate a highly compressed data packet. This highly compressed data packet contains header information for decoding and the compressed data body.

[0035] This system iterates through the initially compressed data blocks, counting the frequency of each distinct run-length value and representative value, and sets a frequency threshold, for example, a value occurring more than once or exceeding 20% ​​of the total number of segments. Values ​​whose frequency reaches or exceeds this threshold are considered "high-frequency occurrences" and are included in the local dictionary to be constructed. The run-length encoding and representative value encoding here specifically refer to the short binary codewords dynamically generated by this algorithm. Their generation follows the variable-length encoding principle in data compression, that is, sorting the values ​​in the selected dictionary from high to low frequency and assigning them unique prefix codes of increasing length, such as similar to canonical Huffman coding, thus assigning shorter codewords to high-frequency values. Before secondary compression, the run-length and representative values ​​exist in memory in a fixed-width original encoding form; for example, the length is represented by a 1-2 byte integer, and the representative value is represented by a 4-byte single-precision floating-point number. This fixed-width data is the original encoding that is subsequently replaced.

[0036] During encoding, the system traverses the initially compressed data blocks, using the original encoding of each segment's length and representative value as the key to search for the corresponding short codeword entry in the constructed local dictionary. If found, the short codeword replaces the original encoding; if no low-frequency value is found, its original encoding is retained, but a specific unique prefix identifier must be appended to it for clear differentiation in the bitstream. Finally, the system packages the encoded codeword sequence, the used local dictionary, and the decoding rule index together. The decoding rule index, as part of the high-compression ratio data packet header, must contain all the metadata necessary for the decoding end to correctly recover the data. This mainly includes: a mapping table between high-frequency values ​​and short codewords in the local dictionary, a description of the original encoding format such as length and bit width of the representative value, and a definition of a specific identifier used to mark low-frequency values.

[0037] The above process illustrates how the improved run-length encoding algorithm dynamically adjusts the segmentation threshold based on the current power consumption pattern. During peak periods, a smaller segmentation threshold coefficient makes the algorithm more sensitive to data fluctuations, generating more and shorter segments to capture power spikes. The adaptive dictionary encoding algorithm then dynamically constructs a local dictionary based on the statistical characteristics of these segments, further compressing the data. The real-time dynamic segmentation threshold T can be calculated using the following formula: in: This indicates the current segmentation threshold. This represents the segmented threshold baseline coefficient determined by the current electricity consumption pattern. This represents the overall range of the original electrical energy data sequence, i.e., the difference between the maximum and minimum values ​​in the sequence. In peak mode, The value is small, resulting in A smaller value makes the conditions for run merging more stringent, allowing for timely segmentation at points of rapid data change; in stable mode, The value is relatively large. A larger value allows for greater numerical fluctuations to be merged into the same run, improving compression efficiency.

[0038] In one embodiment of the present invention, the implementation process of the anomaly screening and dual-queue transmission mechanism is described. The original electrical energy data sequence at the current moment is decoded and reconstructed from a high-compression ratio data packet. The reference electricity consumption curve of the smart meter corresponding to the data packet is retrieved from a locally maintained meter behavior feature library. This reference electricity consumption curve is obtained through statistical learning from the meter's historical normal electricity consumption data for the same period. The dynamic time warping distance between the currently decoded original electrical energy data sequence and the corresponding reference electricity consumption curve is calculated. This dynamic time warping distance is compared with a preset distance threshold. If the calculated dynamic time warping distance exceeds the preset distance threshold, the current data packet is determined to contain potentially abnormal electricity consumption data, and an anomaly mark is added to the high-compression ratio data packet; otherwise, it is considered a normal data packet. For high-compression ratio data packets marked as abnormal, key anomaly summary information is extracted from the decoded original electrical energy data sequence. The extracted information includes the time of anomaly occurrence, the duration of the anomaly, and the magnitude of the anomaly.

[0039] See Figure 3 A dual-queue caching structure is established within the edge computing gateway: a real-time transmission queue for real-time transmission and a batch transmission queue for batch processing. High-compression ratio data packets identified as having anomaly tags, along with their corresponding anomaly summary information, are immediately placed into the real-time transmission queue. High-compression ratio data packets identified as untagged anomalies are categorized and cached in the batch transmission queue according to their corresponding smart meter identifier and timestamp information. The real-time transmission queue uses an event-triggered mechanism; once the queue is not empty, the edge computing gateway initiates immediate single-packet transmission to the main station system. The batch transmission queue uses a time-triggered mechanism; when the accumulated number of untagged anomaly high-compression ratio data packets reaches a preset threshold, or when a fixed aggregation transmission period is reached, the subsequent batch aggregation transmission process is triggered.

[0040] In practical implementation, the edge computing gateway performs outlier screening and marking on high-compression data packets generated by a smart meter. The high-compression data packets are then decoded using agreed-upon decoding rules to recover the original power data sequence. For example, the decoded data sequence, with 15-minute intervals, is obtained within a time window from 2:00 PM to 3:00 PM on a certain afternoon: [0.85, 0.87, 1.20, 5.60, 5.58, 1.10, 0.86, 0.85]. The gateway retrieves the benchmark electricity consumption curve for the smart meter from its locally maintained meter behavior feature library. This library stores a benchmark sequence generated through statistical learning from the meter's historical normal electricity consumption data for the same period. For example, the benchmark electricity consumption curve sequence might be: [0.82, 0.84, 0.86, 0.88, 0.90, 0.87, 0.84, 0.82].

[0041] The dynamic time warping algorithm finds the optimal warping path between two sequences of potentially different lengths by constructing a cumulative distance matrix and calculating the minimum cumulative distance along that path. In this embodiment, the dynamic time warping distance between the currently decoded raw energy data sequence and the corresponding time-based reference electricity consumption curve is calculated. Let this dynamic time warping distance be D, and its calculation can be expressed as: in: Indicates the dynamic time-warped distance. This represents a regular path from the start to the end of a sequence. This represents the total number of steps in the regularized path. This represents the first element in the original electrical energy data sequence S. Data points, This represents the first reference power consumption curve in the sequence T. The data point is on the path W. Step, index of the raw electrical energy data sequence Index of the reference power consumption curve sequence A correspondence is established. In a specific example, the decoded sequence [0.85, 0.87, 1.20, 5.60, 5.58, 1.10, 0.86, 0.85] and the reference sequence [0.82, 0.84, 0.86, 0.88, 0.90, 0.87, 0.84, 0.82] are dynamically time-warped to obtain a quantized distance value. The calculated dynamic time-warped distance The distance is compared with a preset distance threshold, which is set based on the fluctuation range of historical normal data. If the distance is dynamically time-normalized... If the distance exceeds a preset threshold, the current high-compression ratio data packet is determined to contain potentially abnormal power consumption data, and an abnormality marker is added to the high-compression ratio data packet; otherwise, it is considered a normal data packet. In the example, because the decoded sequence shows a significant power increase after the third point, which differs significantly from the stable baseline curve, the calculated... The value is likely to exceed the threshold, so the data packet is marked as abnormal. For high-compression ratio data packets marked as abnormal, key anomaly summary information is extracted from the decoded original electrical energy data sequence. The extracted information includes the time of anomaly occurrence, the duration of anomaly, and the magnitude of anomaly.

[0042] It is understandable that the real-time transmission queue adopts an event-triggered mechanism. Once the real-time transmission queue is not empty, the edge computing gateway initiates immediate single-packet transmission to the main station system. For example, if an abnormal data packet for meter METER-1001 is detected in the real-time transmission queue, it is immediately sent separately through the communication module. The batch transmission queue adopts a time-triggered mechanism. When the accumulated number of unmarked abnormal high-compression ratio data packets reaches a preset threshold, or reaches a fixed aggregation and sending cycle, the subsequent batch aggregation and transmission process is triggered. The combination of the two queues and the two triggering mechanisms realizes a transmission strategy that prioritizes and quickly reports abnormal data and efficiently aggregates and sends normal data later.

[0043] In one embodiment of the present invention, the implementation process of data packet merging in batch aggregation transmission is described. In periodic batch aggregation transmission, high-compression ratio data packets with unmarked anomalies within multiple time windows are merged into an aggregated data block according to a preset aggregation rule. The specific implementation includes: reading a preset number of high-compression ratio data packets with unmarked anomalies from the batch transmission queue in chronological order; parsing the header information of each high-compression ratio data packet to extract its corresponding smart meter identifier, data start time, and data end time; grouping all read high-compression ratio data packets according to the smart meter identifier; within the group of the same smart meter identifier, sequentially concatenating the payload data portions of multiple high-compression ratio data packets according to the order of their data start times; generating a new aggregation header for the concatenated overall data block, which includes the identifiers of all aggregated smart meters, the total time range of the aggregated data, and the offset index of each high-compression ratio data packet within the aggregated data block; and combining the newly generated aggregation header with the concatenated overall data block to form the final aggregated data block.

[0044] In practical implementation, the edge computing gateway's batch transmission queue adopts a time-triggered mechanism. When the accumulated number of unlabeled abnormal high-compression ratio data packets reaches a preset threshold or a fixed aggregation sending period, the batch aggregation transmission process is initiated. During periodic batch aggregation transmission, unlabeled abnormal high-compression ratio data packets from multiple time windows are merged into aggregated data blocks according to preset aggregation rules. A preset number of unlabeled abnormal high-compression ratio data packets are read from the batch transmission queue in chronological order. The preset number can be 100. The edge computing gateway reads 100 high-compression ratio data packets sequentially, starting from the head of the batch transmission queue. The header information of each high-compression ratio data packet is parsed, extracting the smart meter identifier, data start time, and data end time contained in the header information. The header of a high-compression ratio data packet may contain the following fields: meter ID "METER_1001", data start time "2025-10-27 10:00:00", and data end time "2025-10-27 10:15:00" (see Table 1).

[0045] Table 1: Offset Index Table within Aggregated Data Blocks All read high-compression data packets are grouped according to their smart meter identifiers. A container with the smart meter identifier as the key is created in memory, grouping high-compression data packets with the same smart meter identifier into the same group. Within each group with the same smart meter identifier, the payload data portions of multiple high-compression data packets are sequentially concatenated according to their data start time. For example, for a group with the smart meter identifier "METER_1001," which contains two high-compression data packets, one with a data start time of 10:00 and the other of 10:15, the payload portions of these two data packets are concatenated into a continuous data block in chronological order. After sequential concatenation, a new aggregation header is generated for the resulting overall data block. The aggregation header contains the identifiers of all aggregated smart meters, the total time range of the aggregated data, and the offset index of each high-compression data packet within the aggregated data block. The total time range of the aggregated data is the union of the time spans of all aggregated data packets, for example, from the earliest data start time "2025-10-27 10:00:00" to the latest data end time "2025-10-27 10:30:00". The offset index of each high-compression data packet within the aggregated data block indicates the starting position and length of the packet's payload data within the overall data block. The calculation can be expressed as: in: Indicates the current number The starting byte offset of a high-compression packet payload within an aggregated data block. Indicates the fixed length (in bytes) of the aggregate header. Indicates the order in which it is arranged. The first data packet before The length (in bytes) of the high-compression packet payload; the summation sign indicates the summation of the previous data packets. The lengths of each data packet payload are accumulated. The aggregate header itself contains a table recording these offset indices. The aggregate header is combined with the concatenated whole data block to form the final aggregate data block, which has the structure of the aggregate header followed immediately by the concatenated whole data payload.

[0046] In some embodiments, the process of generating the aggregate header includes creating a binary structure. This binary structure begins with a magic number identifying the type of aggregated data block, followed by the version number of the aggregated data block, the total length field of the aggregated data block, the total number of smart meters included, and the start and end timestamps of the aggregated time range. Next is a smart meter index table, where each record contains a hash value of a smart meter identification string, the number of data packets contained in that meter, and a pointer to a detailed list of data packets for that meter. Following the smart meter index table is a detailed list of data packets, which creates a record for each aggregated high-compression data packet, recording the original data start time, the original data end time, and the offset of the payload data within the overall data block. and payload data length This structure allows the master station system to quickly locate the data of a specific meter through the smart meter index table during parsing, and then accurately read the payload of each independent high-compression data packet through the offset information in the data packet details list, thereby efficiently de-aggregating the data.

[0047] In one embodiment of the present invention, the implementation process of error control encapsulation and differentiated quality of service strategy is described. Lightweight error control coding is used to encapsulate and protect aggregated data blocks, generating a checksum-enabled transmission frame. Specifically, cyclic redundancy check (CRC) calculation is performed on the aggregated data blocks to generate a first-level checksum. The aggregated data blocks are divided into multiple data fragments of a fixed size. A checksum is calculated for each data fragment to generate second-level fragment-level checksum information. The first-level checksum, all data fragments, and their corresponding second-level fragment-level checksum information are assembled according to a specific frame format. This specific frame format includes a frame start flag, a length field, an address field, a data type identifier, the aggregated data block, the first-level checksum, the second-level fragment-level checksum information, and a frame end flag. The assembled frame structure is the checksum-enabled transmission frame.

[0048] On the communication link between the edge computing gateway and the main system, differentiated quality of service (QoS) policies are assigned to the real-time lossless transmission channel and the batch transmission frame. Two logical data transmission channels with different priorities are configured in the network protocol stack of the edge computing gateway. Data packets flowing to the real-time lossless transmission channel are marked as high priority, and a high-priority QoS policy is configured. This policy mainly includes: a "best-effort" bandwidth allocation mode, meaning no fixed bandwidth is reserved, and transmission only occurs when network bandwidth is idle; allowing a larger queue scheduling delay limit relative to the high-priority channel, such as 500 milliseconds; and configuring a policy where, when network congestion occurs, its packets are dropped with priority over high-priority packets.

[0049] In practical implementation, lightweight error control coding is used to encapsulate and protect the aggregated data block. Assuming the aggregated data block to be protected is 1024 bytes, the edge computing gateway performs cyclic redundancy check (CRC) calculations on the aggregated data block to generate a first-level checksum. Using a CRC-16 polynomial, a modulo-2 division operation is performed on the 1024-byte aggregated data block to generate a 2-byte CRC checksum. The aggregated data block is then divided into multiple data fragments of a fixed size, which can be set to 64 bytes. The 1024-byte aggregated data block is evenly divided into 16 data fragments, each 64 bytes in size. A checksum is calculated for each data fragment to generate second-level fragment-level checksum information. For a 64-byte data fragment, the values ​​of all bytes within the fragment are summed in binary, and the lower 16 bits of the result are retained as the checksum, generating a total of 16 2-byte fragment-level checksums (see Table 2).

[0050] Table 2: Structure of Transmission Frames with Checksum The first-level checksum, all data fragments, and their corresponding second-level fragment-level checksums are assembled according to a specific frame format. This specific frame format includes a frame start flag, a length field, an address field, a data type identifier, an aggregated data block, a first-level checksum, second-level fragment-level checksums, and a frame end flag. The assembly process begins by writing the frame start flag byte, followed by the length field. The value L of the length field is calculated using the following formula: in: This represents the numerical value of the length field. Indicates the length of the address field (6 bytes). Indicates the length of the data type identifier (1 byte). Indicates the length of the aggregated data block (1024 bytes). Indicates the length of the first-level checksum (2 bytes). This indicates the total length (32 bytes) of the second-level fragmentation check information. The address field, data type identifier, aggregated data block, first-level checksum, and second-level fragmentation check information are written in sequence, and finally the frame end marker is written. The assembled frame structure is the transmission frame with checksum.

[0051] In some embodiments, differentiated quality of service (QoS) policies are assigned to the real-time lossless transmission channel and bulk transmission frames on the communication link between the edge computing gateway and the main station system. Two logical data transmission channels with different priorities are configured in the network protocol stack of the edge computing gateway: one logical channel is identified as "High_Prio_Channel," and the other as "Low_Prio_Channel." Data packets flowing to the real-time lossless transmission channel are marked as high priority, and a high-priority QoS policy is configured. This policy includes a guaranteed minimum bandwidth of 100kbps, a lower queue scheduling latency cap of 50 milliseconds, and a smaller packet loss weight factor. Data streams containing bulk transmission frames are marked as low priority, and a low-priority QoS policy is configured. This policy includes a best-effort bandwidth allocation mode, an allowed queue scheduling latency cap of 500 milliseconds, and priority dropping of low-priority queue packets in the event of network congestion.

[0052] It is understandable that during data transmission, the edge computing gateway routes data packets to the corresponding logical data transmission channel for processing and transmission based on the packet type. When the network protocol stack's sending thread processes the transmission queue, it schedules the high-priority "High_Prio_Channel" channel to ensure its real-time performance. For data packets with anomaly markers retrieved from the real-time transmission queue, the network protocol stack marks them with a high-priority tag and places them in the buffer of the "High_Prio_Channel" channel. For data containing batch aggregated transmission frames, the network protocol stack marks them as low-priority and places them in the buffer of the "Low_Prio_Channel" channel. This differentiated processing ensures that abnormal data can be delivered to the main station system with lower latency and higher reliability.

[0053] In one embodiment of the present invention, the implementation process of decoding and data consistency verification in the master station system is described. After receiving the transmission frame, the master station system performs reverse decoding according to the compression encoding rules agreed upon with the edge computing gateway. Upon receiving the transmission frame, the master station system parses its frame format and identifies the data type identifier. If the data type identifier indicates real-time lossless transmission data, high-compression ratio data packets are directly extracted and decoded according to the agreed dictionary encoding rules and dynamic run-length encoding rules to obtain the original energy data sequence and accompanying anomaly summary information. If the data type identifier indicates batch aggregated transmission data, aggregated data blocks are extracted, the aggregated packet header is parsed, and each independent high-compression ratio data packet is separated according to the offset index. For each independent high-compression ratio data packet, decoding is also performed according to the agreed dictionary encoding rules and dynamic run-length encoding rules to recover the original energy data sequence of each smart meter within different time windows.

[0054] The main station system compares the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway during anomaly screening to complete data consistency verification. Specifically, this includes: for data received and decoded via a real-time lossless transmission channel, the main station system logically verifies the decoded original electrical energy data sequence against the accompanying anomaly summary information to confirm that the features described in the anomaly summary information do indeed exist in the data sequence. For normal data received and decoded via batch aggregation transmission, the main station system seamlessly splices it with anomaly data received from the real-time channel from adjacent time periods of the same meter in the time series, checking the continuity and rationality of the data at the splicing point. The main station system calculates the global statistical characteristics of all recovered data and compares them with the statistical data summaries periodically uploaded by the edge computing gateway to verify whether data distortion or loss occurred during transmission and decompression. If distortion or loss is detected, a data retransmission request is initiated to the corresponding edge computing gateway, and a backup redundancy mechanism is used for data repair.

[0055] In practical implementation, after receiving the transmission frame from the edge computing gateway, the master station system performs reverse decoding according to the compression encoding rules agreed upon with the edge computing gateway. Upon receiving the transmission frame, the master station system parses its frame format and identifies the data type identifier field. If the data type identifier field value is 0x01, it indicates that the frame is real-time lossless transmission data; if the value is 0x02, it indicates that the frame is batch aggregated transmission data. If the data type identifier indicates real-time lossless transmission data, the master station system directly extracts the high-compression ratio data packet from the payload portion of the transmission frame and decodes it according to the agreed dictionary encoding rules and dynamic run-length encoding rules. The decoding process parses the header of the high-compression ratio data packet to obtain the local dictionary and decoding rule index carried within. Using the local dictionary, the codeword sequence is restored to the run-length and representative value sequence. Then, based on the segment boundary information recorded in the segment metadata and the real-time dynamic segmentation judgment threshold used in the dynamic run-length encoding rules, the representative value sequence is restored to the original time-series electrical energy data sequence, while simultaneously extracting the accompanying anomaly summary information. If the data type identifier indicates batch aggregated transmission data, the master station system extracts aggregated data blocks from the transmission frame and parses the aggregated packet header of the aggregated data blocks. Based on the offset index in the aggregated packet header, it separates each independent high-compression ratio data packet. Each independent high-compression ratio data packet is also decoded according to the agreed dictionary encoding rules and dynamic run-length encoding rules to recover the original energy data sequence of each smart meter in different time windows.

[0056] The main station system compares the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway during anomaly screening to complete data consistency verification. For data received and decoded through the real-time lossless transmission channel, the main station system performs logical verification on the decoded original electrical energy data sequence and the accompanying anomaly summary information to confirm that the features described in the anomaly summary information do indeed exist in the data sequence. For example, if the anomaly summary information reports an anomaly of magnitude A at time T, the main station system checks whether there are data points near time point T in the decoded sequence that deviate from the benchmark value by A. For normal data received and decoded through batch aggregation transmission, the main station system seamlessly splices it with the anomaly data received from the real-time channel from adjacent time periods of the same meter in the time series, checking the continuity and rationality of the data at the splicing point, such as checking whether the timestamps are continuous and whether there are unreasonable jumps in the power value at the splicing point.

[0057] In a specific implementation, the specific algorithm steps for the seamless splicing performed by the main station system are as follows: The system prepares the data to be spliced, including the raw energy data sequence with anomaly markers decoded from the real-time lossless transmission channel, and the normal data sequence from the smart meter decoded from the batch aggregation transmission channel. Each data point contains a precise timestamp and power value.

[0058] The system merges and sorts the two data sequences. Based on the start time of the time window covered by the data, the two sequences are logically arranged in chronological order. Typically, the abnormal data sequence and the normal data sequence are adjacent in time. The algorithm calculates the difference between the timestamp of the last data point of the previous sequence and the timestamp of the first data point of the next sequence. If this time difference equals the data acquisition period set by the system, the timestamps are considered continuous. The algorithm calculates the absolute difference between the power values ​​of these two points and compares this difference with a reasonableness threshold. This threshold can be dynamically set based on the fluctuation characteristics of the meter's baseline curve or the historical normal fluctuation range. If the power difference does not exceed the threshold, the data change is considered reasonable. If the verification passes, the system performs a splicing operation, creating a new continuous array or list in memory and storing the data points of the two sorted sequences sequentially, thus logically forming a longer, continuous, and unified time series. This spliced ​​complete sequence can be used for subsequent analysis, storage, or display. If the verification fails, an alarm or data inconsistency handling process is triggered.

[0059] The main system calculates the global statistical characteristics of all recovered data and compares them with the statistical data summaries periodically uploaded by the edge computing gateway. The global statistical characteristics include the average value of all recovered data points. The average value is: in: This represents the average value of all time-series electrical energy data points recovered by the main station system. This indicates the total number of recovered electrical energy data points. This represents the value of the i-th electrical energy data point recovered by the master station system. Simultaneously, the master station system calculates the variance of the recovered data. It then compares the calculated average and variance with the average and variance values ​​recorded in the statistical data summary uploaded by the edge computing gateway in the previous statistical period to verify whether data distortion or loss occurred during transmission and decompression. If inconsistency is found, the master station system initiates a data retransmission request to the corresponding edge computing gateway and uses a backup redundancy mechanism to repair the data. This backup redundancy mechanism can involve interpolating and filling in the gaps using data from the previous valid period.

[0060] In some embodiments, the master station system performs de-aggregation operations on batch aggregated transmission data. Based on the offset index table within the aggregated packet header, the master station system extracts the binary payload of each independent high-compression data packet from the aggregated data block using direct memory access operations. The extraction process involves reading the starting offset address and length fields recorded in the offset index table and copying a specified byte segment from the binary stream of the aggregated data block. It is understood that the dictionary encoding rules and dynamic run-length encoding rules used in the decoding process must be completely consistent with the rules used by the edge computing gateway during encoding. This requires a unified code table construction method and dynamic threshold calculation parameters to be agreed upon during system initialization or in the protocol. Optionally, when calculating global statistical features, the master station system can exclude data points marked as abnormal and calculate the mean and variance only based on normal data to obtain a more stable benchmark for comparison. It is understood that the data consistency verification step is the final step to ensure the integrity and reliability of data from the edge computing gateway to the master station system. Logical verification, sequence splicing verification, and global statistical comparison are combined to perform cross-verification from different dimensions: points, lines, and surfaces. If the verification passes, the master station system will store the recovered time-series electrical energy data in the historical database for use by upper-layer applications; if the verification fails and retransmission and repair fail, the master station system will record the event and generate a data integrity alarm.

[0061] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for data compression and transmission of smart meters in distribution areas based on edge computing, characterized in that, This includes a data compression and transmission method running on an edge computing gateway, and a data restoration method running on a main station system, wherein the data compression and transmission method includes: An edge computing gateway is deployed on the transformer side of each distribution area to periodically receive the raw electrical energy data sequence uploaded by all smart meters under the distribution area; Within the edge computing gateway, the received raw electrical energy data sequence is initially compressed using an improved run-length encoding algorithm. The improved run-length encoding algorithm dynamically adjusts its segmentation threshold based on the current electricity consumption pattern to generate initially compressed data blocks and corresponding segmentation metadata. Based on the segmented metadata, the initially compressed data block is compressed a second time using an adaptive dictionary encoding algorithm. The adaptive dictionary encoding algorithm dynamically constructs a local dictionary based on the statistical characteristics of the data block to generate a high compression ratio data packet. The edge computing gateway maintains a meter behavior feature library locally, which stores the historical electricity consumption features of each smart meter. The meter behavior feature library is used to screen and mark outliers in the high compression ratio data packets. For high-compression ratio data packets marked as abnormal, a real-time lossless transmission channel is initiated; for high-compression ratio data packets not marked as abnormal, a periodic batch aggregation transmission mechanism is adopted. The data restoration method includes: After receiving the transmission frame from the edge computing gateway, the main station system performs reverse decoding according to the agreed compression encoding rules to recover the original time-series electrical energy data. The main station system will compare the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway to complete the data consistency verification.

2. The method for data compression and transmission of smart meters in distribution areas based on edge computing according to claim 1, characterized in that, The improved run-length encoding algorithm dynamically adjusts its segmentation threshold based on the current electricity consumption pattern, including: Statistical feature analysis is performed on the original electrical energy data sequence within the current time window to extract its mean, variance, and distribution characteristics of the difference between adjacent data points; Based on the distribution characteristics of the mean, variance, and difference between adjacent data points, a preset pattern matching table is queried to determine the current electricity consumption mode, which includes a stable mode, a fluctuating mode, a peak mode, and a valley mode. A dynamic segmented threshold reference coefficient is associated with each of the aforementioned electricity consumption modes, wherein the segmented threshold reference coefficients corresponding to the fluctuation mode and the peak mode are less than the segmented threshold reference coefficients corresponding to the steady mode and the valley mode. The product of the segmentation threshold benchmark coefficient and the overall range of the original electrical energy data sequence is used as the real-time dynamic segmentation determination threshold for the current time window. During run-length encoding, when the difference in the numerical change of the continuously read raw electrical energy data is less than the real-time dynamic segmentation determination threshold, they are merged into a run-length segment and recorded with representative value and length information to form the preliminary compressed data block and the segmentation metadata that records the segmentation boundary information.

3. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 2, characterized in that, Based on the segmented metadata, the initially compressed data block is further compressed using an adaptive dictionary encoding algorithm, including: The segmented metadata is analyzed to identify the length distribution characteristics and the range of representative values ​​for each run segment in the preliminary compressed data block. Based on the length distribution characteristics and the range of representative values, a local dictionary suitable for the current preliminary compressed data block is dynamically constructed in memory. The local dictionary contains frequently occurring run-length codes and representative value codes. Traverse the initial compressed data block and match the length and representative value of each run segment with the codewords in the local dictionary; The high-frequency run lengths and representative values ​​that are successfully matched are replaced with the corresponding codewords with shorter code lengths in the local dictionary. For low-frequency run lengths and representative values ​​that do not appear in the local dictionary, their original codes are retained but a specific identifier is added. The encoded codeword sequence, the local dictionary, and the decoding rule index are packaged to generate the high-compression data packet.

4. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 3, characterized in that, The high-compression ratio data packets are screened for and marked for outliers using the meter behavior feature database, including: The original electrical energy data sequence at the current moment is decoded from the high compression ratio data packet; The reference electricity consumption curve of the corresponding smart meter is retrieved from the meter behavior feature library. The reference electricity consumption curve is obtained by statistical learning from historical normal electricity consumption data of the same period. Calculate the dynamic time warping distance between the raw electrical energy data sequence at the current moment and the reference electricity consumption curve at the corresponding moment; The dynamic time-normalized distance is compared with a preset distance threshold; If the dynamic time warping distance exceeds the distance threshold, the current data packet is determined to contain potentially abnormal power consumption data, and an abnormal flag is added to the high compression ratio data packet; otherwise, it is considered a normal data packet. For high compression ratio data packets marked as abnormal, key abnormality summary information is extracted from the original electrical energy data sequence, including the time of abnormality occurrence, duration of abnormality, and magnitude of abnormality.

5. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 4, characterized in that, For the high-compression ratio data packets marked as abnormal, a real-time lossless transmission channel is initiated; for the high-compression ratio data packets not marked as abnormal, a periodic batch aggregation transmission mechanism is employed, including: A dual-queue cache is established inside the edge computing gateway, namely a real-time transmission queue and a batch transmission queue; The high-compression ratio data packets identified as anomalous and their corresponding anomalous summary information are immediately placed into the real-time transmission queue; The high compression ratio data packets identified as unmarked anomalies are categorized and cached in the batch transmission queue according to the identifier and timestamp of their corresponding smart meters. The real-time transmission queue adopts an event-triggered mechanism. Once the queue is not empty, it initiates the instant transmission of a single packet to the main station system. The batch transmission queue adopts a time-triggered mechanism. When the number of accumulated unmarked abnormally high compression ratio data packets reaches a preset threshold or a fixed aggregation sending period is reached, subsequent batch aggregation transmission is triggered.

6. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 5, characterized in that, The method further includes: In periodic batch aggregation transmission, high compression ratio data packets with unmarked anomalies within multiple time windows are merged into aggregated data blocks according to preset aggregation rules; Lightweight error control coding is used to encapsulate and protect the aggregated data block, generating a transmission frame with verification. On the communication link between the edge computing gateway and the main station system, differentiated network service quality policies are allocated to the real-time lossless transmission channel and batch transmission frames to prioritize the bandwidth and low latency of the real-time lossless transmission channel. After receiving the transmission frame, the master station system performs reverse decoding according to the compression encoding rules agreed upon with the edge computing gateway to recover the original time-series electrical energy data; The main station system will compare the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway during anomaly screening to complete the data consistency verification. High-compression data packets with unmarked anomalies from multiple time windows are merged into aggregated data blocks according to preset aggregation rules, specifically including: A predetermined number of high-compression data packets with unmarked anomalies are read from the batch transmission queue in chronological order. Parse the header information of each high-compression data packet to extract its corresponding smart meter identifier, data start time, and data end time; All high-compression data packets are grouped according to the smart meter identifier; Within the same smart meter identifier group, the payload data portions of multiple high-compression ratio data packets are sequentially spliced ​​together according to the order of their data start times. A new aggregation header is generated for the spliced ​​overall data block. The aggregation header contains all smart meter identifiers of the aggregated data, the total time range of the aggregated data, and the offset index of each high-compression data packet within the aggregated data block. The aggregated header is combined with the concatenated overall data block to form the aggregated data block.

7. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 6, characterized in that, The aggregated data block is encapsulated and protected using lightweight error control coding, including: Cyclic redundancy check calculation is performed on the aggregated data block to generate a first-level check code; The aggregated data block is divided into multiple data fragments of a fixed size; Calculate the checksum for each data fragment and generate second-level fragment-level checksum information; The first-level checksum, all data fragments and their corresponding second-level fragment-level checksums are assembled according to a specific frame format, which includes a frame start flag, a length field, an address field, a data type identifier, an aggregated data block, a first-level checksum, second-level fragment-level checksums and a frame end flag. The assembled frame structure is the transmission frame with verification.

8. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 7, characterized in that, Assigning differentiated quality of service policies to the real-time lossless transmission channel and bulk transmission frames, including: In the network protocol stack of the edge computing gateway, two logical data transmission channels with different priorities are configured; Data packets flowing to the real-time lossless transmission channel are marked as high priority, and a high-priority quality of service policy is configured. The high-priority quality of service policy includes guaranteed minimum bandwidth, lower queue scheduling latency, and lower packet loss weight. The data stream containing the batch transmission frames is marked as low priority, and a low-priority quality of service policy is configured, which includes best-effort bandwidth allocation, allowing large scheduling delays, and prioritizing packet drop during congestion. During data transmission, data packets are routed to the corresponding logical data transmission channel for processing and transmission based on their type.

9. The data compression and transmission method for smart meters in distribution areas based on edge computing according to claim 6, characterized in that, After receiving the transmission frame, the master station system performs reverse decoding according to the compression encoding rules agreed upon with the edge computing gateway, including: After receiving the transmission frame, the master station system first parses its frame format and identifies the data type identifier; If the data type identifier indicates real-time lossless transmission data, then the high compression ratio data packet is directly extracted and decoded according to the agreed dictionary encoding rules and dynamic run-length encoding rules to obtain the original electrical energy data sequence and anomaly summary information; If the data type identifier indicates that the data is being transmitted in batches, then the aggregated data blocks are extracted, the aggregated header is parsed, and each independent high-compression data packet is separated according to the offset index. For each independent high-compression data packet, it is also decoded according to the agreed dictionary encoding rules and dynamic run-length encoding rules to recover the original energy data sequence of each smart meter in different time windows; The main station system compares the recovered time-series electrical energy data with the anomaly summary information uploaded by the edge computing gateway during anomaly screening to complete data consistency verification, specifically including: For data received and decoded through a real-time lossless transmission channel, the master station system performs logical verification on the decoded raw electrical energy data sequence and the accompanying anomaly summary information to confirm that the features described in the anomaly summary information do indeed exist in the data sequence. For normal data received and decoded through batch aggregation transmission, the main station system seamlessly splices it with abnormal data received from the real-time channel from adjacent time periods of the same electricity meter in the time series, and checks the continuity and rationality of the data at the splicing point. The main station system calculates the global statistical characteristics of all recovered data and compares them with the statistical data summaries periodically uploaded by the edge computing gateway to verify whether the data has been distorted or lost during transmission and decompression. If the verification finds an inconsistency, a data retransmission request is initiated to the corresponding edge computing gateway, and the data is repaired using a backup redundancy mechanism.

10. A data compression and transmission system for smart meters in distribution areas based on edge computing, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the edge computing-based smart meter data compression and transmission method according to any one of claims 1 to 9.

Images