Multi-feature adaptive data compression method and system for power distribution network terminal

Abstract

The invention discloses a multi-feature adaptive data compression method and system for a power distribution network terminal, and the method comprises the steps: extracting a multi-dimensional feature vector of to-be-compressed data, inputting the multi-dimensional feature vector into a classification module, and obtaining the type of the to-be-compressed data; determining a candidate compression algorithm type according to the type of the to-be-compressed data; calculating the score of each compression algorithm instance in the candidate compression algorithm type to the to-be-compressed data; and selecting the compression algorithm instance with the highest score value to compress the to-be-compressed data. According to the method, fine classification is carried out on the power distribution network data in consideration of multiple dimensions, the optimal compression algorithm is selected in combination with data features, and the service requirements, the compression effect and the compression efficiency are effectively considered.

Description

Technical Field

[0001] This invention relates to a multi-feature adaptive data compression method and system for distribution network terminals, belonging to the field of distribution network data transmission technology. Background Technology

[0002] Currently, distribution network data is characterized by its diverse types, large volume, and high real-time requirements. Directly transmitting or storing this massive amount of data would lead to information transmission delays, slow access speeds, and impact on system real-time performance. Therefore, data compression is necessary. Chinese patent application CN119093942A discloses a data compression method for compressing load power consumption data. This method divides the data into segments based on fluctuation levels and uses different compression algorithms for different segments, balancing compression accuracy and efficiency. However, this method's data segmentation is relatively coarse, resulting in a limited selection of compression algorithms. It cannot handle the highly heterogeneous distribution network data, which exhibits significant differences in structure, accuracy requirements, real-time performance, and physical characteristics. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multi-feature adaptive data compression method for distribution network terminals, which addresses the shortcomings of the existing technology. This method takes into account multiple dimensions to perform fine classification of distribution network data, selects the optimal compression algorithm based on data features, and effectively balances business needs, compression effect and compression efficiency.

[0004] To address the aforementioned technical problems, this invention discloses a multi-feature adaptive data compression method for distribution network terminals, comprising the following steps:

[0005] Extract the multidimensional feature vector of the data to be compressed, and input the multidimensional feature vector into the classification module to obtain the type of the data to be compressed;

[0006] Determine the candidate compression algorithm type based on the type of data to be compressed;

[0007] Calculate the score of each compression algorithm instance in the candidate compression algorithm type for the data to be compressed;

[0008] The compression algorithm instance with the highest score is selected to compress the data to be compressed.

[0009] Furthermore, the multidimensional feature vector includes stationary features, periodic features, mutation features, importance features, and real-time features.

[0010] Furthermore, the classification module is a machine learning-based classifier. The input of the classification module is a multi-dimensional feature vector of the data acquired by the distribution network terminal, and the output is a data type label of the data acquired by the distribution network terminal. The classification module is trained by a training set, and the training samples in the training set include the multi-dimensional feature vector and data type label of the historical data acquired by the distribution network terminal.

[0011] Furthermore, determining the candidate compression algorithm type based on the type of data to be compressed specifically involves: querying the compression algorithm type corresponding to the type of data to be compressed in a preset compression algorithm mapping table to obtain the candidate compression algorithm type; the compression algorithm mapping table is used to record data types and corresponding compression algorithm types.

[0012] Furthermore, the calculation of the score for each compression algorithm instance in the candidate compression algorithm type on the data to be compressed specifically includes:

[0013] The compression algorithm library is queried to obtain the estimated compression ratio, estimated reconstruction error, and estimated computation time of each compression algorithm instance in the candidate compression algorithm type; the compression algorithm library is used to store the parameter values, estimated compression ratio, estimated reconstruction error, and estimated computation time of the compression algorithm instances in the compression algorithm type.

[0014] Calculate the score of the data to be compressed for each compression algorithm instance:

[0015]

[0016] in The score for the i-th compression algorithm instance among the candidate compression algorithm types. , , These are the estimated compression ratio, estimated reconstruction error, and estimated computation time for the i-th compression algorithm instance among the candidate compression algorithm types, respectively. , , These are weighting coefficients, all constants between 0 and 1, and ; , , These represent the target compression ratio, maximum permissible error, and maximum permissible delay of the data to be compressed.

[0017] Furthermore, the compression algorithm library uses a list to record the parameter values, estimated compression ratio, estimated reconstruction error, and estimated computation time of each compression algorithm instance in the compression algorithm type. The compression algorithm library is stored in the non-volatile memory of the processor chip that executes the compression algorithm.

[0018] Furthermore, it also includes:

[0019] Calculate the transmission priority of the compressed data; add the compressed data to transmission queues of different priorities according to the transmission priority.

[0020] Furthermore, it also includes:

[0021] After the receiving end decompresses the received compressed data, it calculates the reconstruction error.

[0022] The reconstruction error is estimated by updating the corresponding compression algorithm instance in the compression algorithm library based on the statistically obtained reconstruction error.

[0023] Furthermore, it also includes:

[0024] Channel quality is obtained by probing frames, and the target compression ratio of the data to be compressed is adjusted according to the channel quality.

[0025] On the other hand, the present invention also discloses a multi-feature adaptive data compression system for distribution network terminals, comprising:

[0026] Multidimensional feature vector calculation module 1 is used to extract multidimensional feature vectors from the data to be compressed.

[0027] Classification module 2 is used to obtain the type of data to be compressed based on the multidimensional feature vector of the data to be compressed;

[0028] Candidate compression algorithm type determination module 3 determines the candidate compression algorithm type based on the type of data to be compressed;

[0029] The scoring calculation module 4 is used to calculate the score of each compression algorithm instance in the candidate compression algorithm type for the data to be compressed;

[0030] Data compression module 5 is used to select the compression algorithm instance with the highest score to compress the data to be compressed.

[0031] Beneficial effects: The multi-feature adaptive data compression method and system for distribution network terminals disclosed in this invention have the following advantages:

[0032] 1. Using multidimensional feature vectors to perform refined classification of distribution network data, multidimensional feature vectors can accurately characterize the intrinsic characteristics of different data sequences, significantly improve the accuracy of data type identification, and provide a quantitative basis for subsequent compression algorithm matching;

[0033] 2. By adopting a two-stage matching method that first determines the candidate compression algorithm type and then determines the compression algorithm instance, the system takes into account both data type and business requirements, and can achieve an adaptive balance between compression accuracy, efficiency and business needs.

[0034] 3. The determination of weight values ​​and target compression ratios is combined with real-time data characteristics and dynamic adjustment of channel conditions to predict and comprehensively score the performance of candidate algorithms, dynamically select the optimal compression algorithm and parameters, realize adaptive compression of "applying pressure according to type", and can be dynamically adjusted according to channel congestion.

[0035] 4. By utilizing the actual reconstruction error fed back from the receiving end, the performance of the compression algorithm instance is dynamically updated, guiding subsequent algorithm selection and parameter optimization. This mechanism forms a closed loop of "compression-feedback-optimization," enabling the compression strategy to evolve online according to changes in data characteristics and system operating status, continuously maintaining optimal compression performance and avoiding performance degradation caused by environmental changes or data distribution drift.

[0036] 5. The transmission of compressed data adopts a priority-aware queue scheduling method. By comprehensively considering the importance of data, waiting time, and urgency coefficient to calculate the transmission priority, it ensures that high-value data (such as fault events) are transmitted with low latency, and can still guarantee the real-time performance of critical services when bandwidth is limited, thus achieving the optimal allocation of limited channel resources. Attached Figure Description

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0038] Figure 1 This is a flowchart of the multi-feature adaptive data compression method for distribution network terminals disclosed in this invention;

[0039] Figure 2 This is a schematic diagram of the composition of a multi-feature adaptive data compression system for distribution network terminals. Detailed Implementation

[0040] There are many types of distribution network terminals and a wide variety of data collected. Table 1 provides examples and typical characteristics of several types of meter data.

[0041] Table 1 Example of electricity meter data

[0042] Types of electricity meter data Data item example Typical characteristics Electricity data Positive active energy, energy at various tariff rates, and maximum demand Monotonically increasing / slowly changing AC analog quantity Voltage, current, active power, reactive power, power factor, frequency Cyclical fluctuations and sudden changes Operating data Clock, parameter modification records, power outage and restoration events Discrete state, sudden change Event Log Loss of pressure, phase loss, overload, and excessive demand. Sparse events, high importance Power quality Harmonic content, interharmonics, and three-phase imbalance Quasi-steady state, slow change

[0043] Different compression algorithms are required for different types of data, depending on business needs such as data importance and real-time requirements. This invention discloses a multi-feature adaptive data compression method for distribution network terminals, compressing data collected by devices such as smart meters and concentrators. Figure 1 As shown, the steps include:

[0044] S1. Extract the multidimensional feature vector of the data to be compressed, and input the multidimensional feature vector into the classification module to obtain the type of the data to be compressed;

[0045] In this embodiment, the distribution network data is divided into five types: stable type, periodic type, mutation type, event type, and instruction type. The classification of the distribution network data is based on its multi-dimensional feature vector. In the present invention, the multi-dimensional feature vector includes stability feature, periodicity feature, mutation feature, importance feature, and real-time feature. The specific calculation is as follows:

[0046] The data to be compressed is sampled in the time dimension to form a data sequence with a length of N. The following features are calculated for the data sequence:

[0047] (1) The stability feature combines the variance and the mean of the first-order difference of the data sequence. Specifically:

[0048] Calculate the variance of the data sequence , the mean of the first-order difference . The stability feature S is:

[0049]

[0050] where are the normalization coefficients of different eigenvalues, generally determined through historical records; log is the natural logarithm operation with the natural constant e as the base.

[0051] (2) The periodicity feature is characterized by the maximum autocorrelation coefficient of the data sequence;

[0052] (3) The mutation feature is characterized by detecting whether there are mutation points exceeding the threshold and their density in the sequence. The specific calculation steps are as follows:

[0053] Set a sliding window with a window length of M, where M < N. Starting from the starting point of the data sequence, slide the window. Calculate the mean of the first half of the window length and the mean of the second half of the window length at the current position of the sliding window respectively;

[0054] If , mark that there is one mutation point in this window length. Traverse the entire data sequence and count the number of mutation points C; is the standard deviation of the data sequence, is the sensitivity coefficient. The larger the value, the higher the mutation point judgment threshold and the fewer the obtained mutation points. In this embodiment, it is set to 2;

[0055] Calculate the mutation density: ;

[0056] The window length M of the sliding window affects the number of mutation points C, which can be set according to the data sequence length N, such as using the general formula M=min(max(20,N / 20),200). In this embodiment, the value of M is set according to Table 2 based on different application scenarios.

[0057] Table 2 Window Length Settings for Sliding Windows

[0058] Data length N Window length M Applicable Scenarios Less than 100 points 10~20 Event Analysis 100~1000 points 20~100 Daily electricity meter data Greater than 1000 points 100~200 Monthly data

[0059] (4) Importance characteristics: The value of importance characteristic I is determined according to the purpose of the data to be compressed. In this embodiment, importance is defined as 5 levels, and the relationship between importance characteristics and data purpose is set according to Table 3.

[0060] Table 3. Definition of Importance Characteristics of Data Purpose (I = Level 1-5)

[0061] Data usage Importance features Event log (loss of voltage, phase loss) 5 Operating data (power outage and restoration) 4 Power quality (harmonics) 3 Voltage, current 2 Electricity data 1

[0062] (5) Real-time characteristics, characterized by the maximum allowable delay time of the data service requirements to be compressed.

[0063] The five features obtained above are combined to form a multidimensional feature vector F: ;

[0064] Inputting F into the trained classification module yields the type label K of the data to be compressed. In this embodiment, K can take five values: stationary class K=1, periodic class K=2, mutation class K=3, event class K=4, and instruction class K=5.

[0065] In this embodiment of the invention, the classification module employs a Gaussian Naive Bayes classifier. The classification module is trained offline using a training set comprised of historical data. The training samples in the training set are constructed as follows:

[0066] Data from distribution network terminals at different times is collected. The collected data is sampled along the time dimension to form historical data sequences. Importance features and real-time features are set according to the type of historical data and business needs, and type labels K are assigned according to their purpose. For each historical data sequence, stationarity features, periodicity features, and mutation features are calculated. These, along with the importance features and real-time features, form an original multidimensional feature vector. The feature values ​​of each dimension in all original multidimensional feature vectors are normalized to values ​​in the interval [0, 1], forming the multidimensional vector features corresponding to the historical data sequence. These features, combined with the type label K, constitute training samples.

[0067] For positive active energy and energy data at various tariff rates collected during periods without faults or emergencies, the data is labeled as stable (K=1); for AC analog quantities such as voltage and current collected during periods without faults or emergencies, the data is labeled as periodic (K=2); for energy data or AC analog quantities collected during periods of faults or emergencies, the data is labeled as abrupt change (K=3); for data such as status changes, alarms, abnormal situation records, or operation records uploaded from the terminal to the master station, the data is labeled as event (K=4); and for data such as control commands, parameter setting instructions, setpoint modification instructions, and remote operation instructions issued from the master station to the terminal, the data is labeled as instruction (K=5).

[0068] The Gaussian Naive Bayes classifier is trained offline using training samples to obtain the classification module in this embodiment of the invention, which is then used to classify and predict the type label K of the data sequence to be compressed.

[0069] S2. Determine the candidate compression algorithm type based on the type of data to be compressed;

[0070] Different types of data have different compression requirements. For example, stationary data can be compressed using lossy compression algorithms, while instruction-type data needs to be passed through to ensure real-time performance. In this embodiment, a preset compression algorithm mapping table is used to determine the compression algorithm type based on the classification prediction value K of the data sequence to be compressed by the classification module. The preset compression algorithm mapping table is shown in Table 4.

[0071] Table 4 Compression Algorithm Mapping Table

[0072] K type Compression method type 1 Stable class Improved revolving door 2 Periodic FFT + Quantization 3 Mutant Shift storage 4 Event Class LZ77 5 Instruction class Pass-through

[0073] The compression algorithm type corresponding to the type of data to be compressed is queried in the preset compression algorithm mapping table to obtain the candidate compression algorithm type; the compression algorithm mapping table is used to record the data type and the corresponding compression algorithm type.

[0074] The compression algorithm mapping table only indicates the type of compression algorithm for different K values. For a given class of compression algorithms, different parameter values ​​result in different compression accuracy and time. For example, for the improved revolving door algorithm, key parameters include the threshold value E and the maximum interval T. Different combinations of parameters E and T constitute specific instances of this type of algorithm. In other words, the K value in the compression algorithm mapping table can only determine the type of algorithm used, i.e., the candidate compression algorithm type, but not the specific algorithm instance used.

[0075] S3. Calculate the score of each compression algorithm instance in the candidate compression algorithm types for the data to be compressed; specifically including:

[0076] The compression algorithm library is queried to obtain the estimated compression ratio, estimated reconstruction error, and estimated computation time of each compression algorithm instance in the candidate compression algorithm type; the compression algorithm library is used to store the parameter values, estimated compression ratio, estimated reconstruction error, and estimated computation time of the compression algorithm instances in the compression algorithm type.

[0077] Calculate the score of the data to be compressed for each compression algorithm instance:

[0078]

[0079] in The score for the i-th compression algorithm instance among the candidate compression algorithm types. , , These are the estimated compression ratio, estimated reconstruction error, and estimated computation time for the i-th compression algorithm instance among the candidate compression algorithm types, respectively. , , These are weighting coefficients, all constants between 0 and 1, and ; , , These represent the target compression ratio, maximum permissible error, and maximum permissible delay of the data to be compressed.

[0080] The compression ratio is estimated using the theoretical compression ratio of the compression algorithm under different parameters; the initial value of the reconstruction error can be estimated using the theoretical error value; and the computation time is estimated using the theoretical complexity of different compression algorithms.

[0081] In this embodiment, the estimated reconstruction error is updated based on the measured reconstruction error, making the selected compression algorithm more adaptable to actual conditions. After sending multiple compressed data blocks, the sending end inserts a complete data block. The receiving end compares the decompressed data and the original data at that moment to calculate the reconstruction error. Different error thresholds are selected for different types of data; if the error exceeds the threshold, the calculated reconstruction error is fed back to the sending end. The sending end uses the received reconstruction error to overwrite the compression algorithm instance's score in subsequent score calculations, resulting in a lower score for the corresponding compression algorithm instance.

[0082] The compression algorithm library uses a list to record the parameter values, estimated compression ratio, estimated reconstruction error, and estimated computation time of each compression algorithm instance in the compression algorithm type. In order to speed up the processing, the compression algorithm library is stored in the non-volatile memory of the processor chip that executes the compression algorithm.

[0083] Target compression ratio of the data to be compressed A fixed value can be set according to business requirements, or it can be determined based on channel quality. Execution occurs at the initial moment or during a timed period. The receiver obtains channel quality information, such as receive gain, signal-to-noise ratio (SNR), and time slot allocation, through probe frames. The target compression ratio of the data to be compressed is then adjusted based on the channel quality. A smaller target compression ratio is set when channel quality is good, resulting in larger compressed data, allowing for faster transmission using the better channel. Conversely, a larger target compression ratio is set when channel quality is poor, resulting in smaller compressed data, thus achieving faster transmission. In other words, the compression ratio is increased during congestion to ensure throughput, and decreased during idle periods to ensure accuracy.

[0084] Maximum permissible error The size is determined based on the importance of the business; more important businesses have smaller settings. Value; Maximum allowable delay Determined based on business requirements, for example, a requirement for millisecond-level latency in response. Set to 10ms. If fault warning requires seconds, set to 1s. For regular business, allow for minute-level delays, which can be set to 1min, 5mins, 10mins, etc., depending on the business.

[0085] Weighting coefficient , , The value can be adjusted according to actual business needs; a high compression ratio... Increased data accuracy requirements Increased real-time requirements Increase. Table 5 shows the recommended weight values ​​for different scenarios.

[0086] Table 5 Recommended values ​​for weighting coefficients

[0087] Scene w1 w2 w3 Channel bandwidth limited or channel congestion 0.7 0.2 0.1 High precision requirements 0.1 0.8 0.1 Real-time control 0.1 0.2 0.7 conventional 0.4 0.3 0.3

[0088] S4. Select the compression algorithm instance with the highest score to compress the data to be compressed.

[0089] The compressed data forms a compressed package for transmission or storage. In this embodiment of the invention, a priority is set for the compressed packages used for transmission, and data is transmitted according to the priority to ensure that important and urgent data can be sent first. Specifically:

[0090] Calculate the transmission priority P of the compressed data:

[0091]

[0092] Where I represents the importance feature in the multidimensional feature vector of the data; This represents the data waiting time, and Urgency is the urgency factor. A higher value indicates greater urgency; for example, a fault event is 1.0, and regular data is 0.1. This is the transmission priority weighting coefficient, which is set according to the field application. In this embodiment, it is set as follows: This means that an important feature is promoted by one level, and its priority is increased by one. Real-time performance is measured in seconds; data is generated after... Initially set to 1, its priority decays as the event wait time increases, preventing old data from consuming new data resources; For example, in the case of a fault event, the priority contribution is 100, requiring priority processing; the priority contribution for a normal value is 1, indicating that the data is not urgent. Compressed data is inserted into transmission queues of different priorities according to the P value for transmission.

[0093] This invention also discloses a multi-feature adaptive data compression system for distribution network terminals, such as... Figure 2 As shown, it includes:

[0094] Multidimensional feature vector calculation module 1 is used to extract multidimensional feature vectors from the data to be compressed.

[0095] Classification module 2 is used to obtain the type of data to be compressed based on the multidimensional feature vector of the data to be compressed;

[0096] Candidate compression algorithm type determination module 3 determines the candidate compression algorithm type based on the type of data to be compressed;

[0097] The scoring calculation module 4 is used to calculate the score of each compression algorithm instance in the candidate compression algorithm type for the data to be compressed;

[0098] Data compression module 5 is used to select the compression algorithm instance with the highest score to compress the data to be compressed.

[0099] This invention provides a concept and method for multi-feature adaptive data compression for distribution network terminals. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A multi-feature adaptive data compression method for distribution network terminals, characterized in that, Including the following steps: Extract the multidimensional feature vector of the data to be compressed, and input the multidimensional feature vector into the classification module to obtain the type of the data to be compressed; Determine the candidate compression algorithm type based on the type of data to be compressed; Calculate the score of each compression algorithm instance in the candidate compression algorithm type for the data to be compressed; The compression algorithm instance with the highest score is selected to compress the data to be compressed.

2. The multi-feature adaptive data compression method according to claim 1, characterized in that, The multidimensional feature vector includes stationary features, periodic features, mutation features, importance features, and real-time features.

3. The multi-feature adaptive data compression method according to claim 2, characterized in that, The classification module is a machine learning-based classifier. The input of the classification module is the multidimensional feature vector of the data acquired by the distribution network terminal, and the output is the data type label of the data acquired by the distribution network terminal. The classification module is trained by a training set, and the training samples in the training set include the multidimensional feature vector and data type label of the historical data acquired by the distribution network terminal.

4. The multi-feature adaptive data compression method according to claim 1, characterized in that, The step of determining the candidate compression algorithm type based on the type of data to be compressed specifically involves: querying the compression algorithm type corresponding to the type of data to be compressed in a preset compression algorithm mapping table to obtain the candidate compression algorithm type; the compression algorithm mapping table is used to record data types and corresponding compression algorithm types.

5. The multi-feature adaptive data compression method according to claim 1, characterized in that, The calculation of the score for each compression algorithm instance in the candidate compression algorithm type on the data to be compressed specifically includes: The compression algorithm library is queried to obtain the estimated compression ratio, estimated reconstruction error, and estimated computation time of each compression algorithm instance in the candidate compression algorithm type; the compression algorithm library is used to store the parameter values, estimated compression ratio, estimated reconstruction error, and estimated computation time of the compression algorithm instances in the compression algorithm type. Calculate the score of the data to be compressed for each compression algorithm instance: ； in The score for the i-th compression algorithm instance among the candidate compression algorithm types. , , These are the estimated compression ratio, estimated reconstruction error, and estimated computation time for the i-th compression algorithm instance among the candidate compression algorithm types, respectively. , , These are weighting coefficients, all constants between 0 and 1, and ; , , These represent the target compression ratio, maximum permissible error, and maximum permissible delay of the data to be compressed.

6. The multi-feature adaptive data compression method according to claim 5, characterized in that, The compression algorithm library uses a list to record the parameter values, estimated compression ratio, estimated reconstruction error, and estimated computation time of each compression algorithm instance in the compression algorithm type. The compression algorithm library is stored in the non-volatile memory of the processor chip that executes the compression algorithm.

7. The multi-feature adaptive data compression method according to claim 1, characterized in that, Also includes: Calculate the transmission priority of the compressed data; The compressed data is added to transmission queues of different priorities according to the transmission priority.

8. The multi-feature adaptive data compression method according to claim 1, characterized in that, Also includes: After the receiving end decompresses the received compressed data, it calculates the reconstruction error. The reconstruction error is estimated by updating the corresponding compression algorithm instance in the compression algorithm library based on the statistically obtained reconstruction error.

9. The multi-feature adaptive data compression method according to claim 1, characterized in that, Also includes: Channel quality is obtained by probing frames, and the target compression ratio of the data to be compressed is adjusted according to the channel quality.

10. A multi-feature adaptive data compression system for distribution network terminals, characterized in that, include: The multidimensional feature vector calculation module (1) is used to extract the multidimensional feature vector of the data to be compressed; The classification module (2) is used to obtain the type of the data to be compressed based on the multidimensional feature vector of the data to be compressed; Candidate compression algorithm type determination module (3) determines the candidate compression algorithm type based on the type of data to be compressed; The scoring calculation module (4) is used to calculate the score of each compression algorithm instance in the candidate compression algorithm type for the data to be compressed; The data compression module (5) is used to select the compression algorithm instance with the highest score to compress the data to be compressed.

Images