Method, device and equipment for compressing power system historical data and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YANTAI HAIYI SOFTWARE
- Filing Date
- 2026-05-07
- Publication Date
- 2026-08-07
AI Technical Summary
[0003]现有技术中的电力系统历史数据压缩方法通常采用通用压缩算法，未能充分利用电力系统历史数据的时序规律性和数值平稳性特征，导致压缩效率较低、存储资源占用较大
[0043]本申请提供一种电力系统历史数据的压缩方法，包括：从电力系统数据采集终端获取电力系统历史数据集合，所述历史数据集合包括时序标识信息和关联数据值；对所述历史数据集合进行异常数据检测，剔除异常数据；对剔除异常数据后的历史数据集合进行缺失数据补全；根据所述时序标识信息的先后顺序对补全后的历史数据集合进行排序，选取排序后首个时序点的时序标识作为基准时序点；基于所述基准时序点，计算排序后各时序点与所述基准时序点的时间跨度差值；根据所述时间跨度差值的数值范围，将所述时间跨度差值划分为多个编码层级，按编码层级对所述时间跨度差值进行二进制编码，构建时序骨架；对所述时序骨架中的各时序点，如果与前一时刻的时序点之间的时间间隔落入所述时序骨架中对应编码层级的时间跨度范围内，且所述时序点的关联数据值与所述前一时刻的时序点的关联数据值完全一致，则将所述时序点标记为重复数据，保留首个时序点的完整数据并存储二进制标识替代后续重复时序点，否则，维持所述时序点的完整数据；对未标记为重复数据的关联数据值，计算与前一有效时序点的对应维度关联数据值之间的差值作为差分结果；根据所述差分结果的数值范围，对所述差分结果进行自适应存储；根据所述时序骨架的编码结果和所述自适应存储后的压缩结果，生成最终压缩数据包。本申请通过将时序分层编码与差分压缩及重复标记相结合，使得时序编码的功能从常规的数据表示转变为支撑数据值压缩算法有效运行的时序框架，时序骨架为差分压缩提供采样间隔可预期性约束，重复标记使差分结果为零的概率增加，三者配合实现压缩效率提高。
Smart Images
Figure CN122533588A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of distribution radio station data characterization, and in particular relates to a method, apparatus, equipment and storage medium for compressing historical data of power systems. Background Technology
[0002] With the continuous advancement of smart grid construction, the amount of data generated by the power system is experiencing explosive growth.
[0003] Existing methods for compressing historical power system data typically employ general compression algorithms, which fail to fully utilize the temporal regularity and numerical stationarity of historical power system data, resulting in low compression efficiency and high storage resource consumption. Summary of the Invention
[0004] The purpose of this application is to overcome the deficiencies in the prior art and provide a method, apparatus, device and storage medium for compressing historical data of a power system.
[0005] This application provides a method for compressing historical data of a power system, including:
[0006] The historical data set of the power system is obtained from the power system data acquisition terminal. The historical data set includes time sequence identification information and associated data values.
[0007] Anomaly detection is performed on the historical data set, and abnormal data is removed;
[0008] Complete the missing data in the historical dataset after removing outliers;
[0009] The completed historical data set is sorted according to the chronological order of the time sequence identifier information, and the time sequence identifier of the first time sequence point after sorting is selected as the reference time sequence point.
[0010] Based on the baseline time series point, calculate the time span difference between each sorted time series point and the baseline time series point;
[0011] Based on the numerical range of the time span difference, the time span difference is divided into multiple coding levels, and the time span difference is binary encoded according to the coding level to construct a time sequence skeleton;
[0012] For each time point in the time sequence skeleton, if the time interval between it and the previous time point falls within the time span range of the corresponding coding level in the time sequence skeleton, and the associated data value of the time point is completely consistent with the associated data value of the previous time point, then the time point is marked as duplicate data, the complete data of the first time point is retained and a binary identifier is stored to replace the subsequent duplicate time points; otherwise, the complete data of the time point is maintained.
[0013] For associated data values that are not marked as duplicates, the difference between them and the corresponding dimension associated data values of the previous valid time series point is calculated as the difference result.
[0014] Based on the numerical range of the difference results, the difference results are adaptively stored;
[0015] Based on the encoding result of the temporal skeleton and the compression result after adaptive storage, the final compressed data packet is generated.
[0016] Optionally, anomaly detection is performed on the historical data set to remove abnormal data, including:
[0017] The system employs a dual criterion of mean and standard deviation and an industry threshold. If the value of the associated data at any time point exceeds the sum of the mean and three times the standard deviation of the associated data or is lower than the difference between the mean and three times the standard deviation, and also exceeds the industry threshold, then the data at any time point is determined to be abnormal and is removed.
[0018] Otherwise, maintain the complete data for any of the stated time points.
[0019] Optionally, missing data can be filled into the historical dataset after removing outlier data, including:
[0020] If there are valid time series points before and after the missing data point, then linear interpolation is used to complete the missing data point by using the associated data values of the valid time series points before and after the missing data point.
[0021] If the missing data point is the first or last data point, then the associated data value of the adjacent valid time series point is used for one-way filling;
[0022] An alarm is triggered if the number of consecutive missing data points exceeds the preset alarm threshold.
[0023] Optionally, based on the numerical range of the time span difference, the time span difference is divided into multiple coding levels, and the time span difference is binary-encoded according to the coding levels to construct a time-series skeleton, including:
[0024] The multiple encoding levels include four levels: the first level has a time span difference range of 0 to 127 milliseconds and uses 8-bit encoding; the second level has a time span difference range of 128 to 4095 milliseconds and uses 14-bit encoding; the third level has a time span difference range of 4096 to 65535 milliseconds and uses 19-bit encoding; and the fourth level has a time span difference range greater than 65535 milliseconds and uses 36-bit encoding.
[0025] Optionally, the repeated data is a data segment in which a predetermined number of consecutive time points all have completely identical associated data values and the time intervals all fall within the time span range of the corresponding encoding level.
[0026] Optionally, for associated data values not marked as duplicate data, the difference between them and the corresponding dimension associated data value of the previous valid time series point is calculated as the difference result, including:
[0027] Before calculating the difference result, a dynamic dictionary is constructed. The initial entries of the dynamic dictionary include industry standard parameters and typical historical data of the equipment. A unique data identifier is assigned to each entry in the dynamic dictionary.
[0028] For associated data values that are not marked as duplicate data, dictionary matching is first performed according to multi-dimensional combinations. If a match is successful, the corresponding data identifier is stored. If the multi-dimensional combination matching fails, dictionary matching is performed one by one according to a single dimension. If a match is successful, the corresponding data identifier is stored. If no matching is successful for any dimension, the difference between the associated data value and the corresponding dimension of the previous valid time series point is calculated as the difference result.
[0029] Optionally, the construction of the dynamic dictionary library includes:
[0030] A sliding window is used to count the frequency of occurrence of the associated data value. If the frequency of occurrence of the associated data value is greater than or equal to a preset frequency threshold and is not in the dynamic dictionary, the associated data value is added to the dynamic dictionary and a data identifier is assigned.
[0031] If the number of entries in the dynamic dictionary reaches the preset capacity limit, the entry with the lowest frequency of occurrence will be discarded.
[0032] This application also provides a multimodal fusion characterization device for transformer area assessment, comprising:
[0033] The acquisition module obtains a set of historical power system data from the power system data acquisition terminal. The set of historical data includes time sequence identification information and associated data values.
[0034] The correction module performs anomaly detection on the historical data set, removes anomaly data, and completes the missing data in the historical data set after removing anomaly data.
[0035] The sorting module sorts the completed historical data set according to the chronological order of the time sequence identifier information, and selects the time sequence identifier of the first time sequence point after sorting as the reference time sequence point.
[0036] The time difference module calculates the time span difference between each sorted time series point and the reference time series point based on the reference time series point;
[0037] The skeleton module divides the time span difference into multiple coding levels according to the numerical range of the time span difference, and performs binary encoding on the time span difference according to the coding level to construct a time-series skeleton.
[0038] The judgment module determines that for each time point in the time sequence skeleton, if the time interval between it and the previous time point falls within the time span range of the corresponding coding level in the time sequence skeleton, and the associated data value of the time point is completely consistent with the associated data value of the previous time point, then the time point is marked as duplicate data, the complete data of the first time point is retained and a binary identifier is stored to replace the subsequent duplicate time points; otherwise, the complete data of the time point is maintained.
[0039] The results module calculates the difference between the associated data values that are not marked as duplicate data and the corresponding dimension associated data values of the previous valid time series point as the difference result; adaptively stores the difference result according to the numerical range of the difference result; and generates the final compressed data packet according to the encoding result of the time series skeleton and the compression result after adaptive storage.
[0040] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0041] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the above-described method.
[0042] The beneficial effects of this application are:
[0043] This application provides a method for compressing historical power system data, comprising: acquiring a historical power system data set from a power system data acquisition terminal, the historical data set including time sequence identifier information and associated data values; performing abnormal data detection on the historical data set and removing abnormal data; completing missing data in the historical data set after removing abnormal data; sorting the completed historical data set according to the chronological order of the time sequence identifier information, and selecting the time sequence identifier of the first time sequence point after sorting as a reference time sequence point; calculating the time span difference between each time sequence point after sorting and the reference time sequence point based on the reference time sequence point; dividing the time span difference into multiple encoding levels according to the numerical range of the time span difference, and performing binary encoding on the time span difference according to the encoding level to construct a time... The temporal skeleton is used for the following process: For each temporal point in the temporal skeleton, if the time interval between it and the previous temporal point falls within the time span of the corresponding coding level in the temporal skeleton, and the associated data value of the temporal point is completely consistent with the associated data value of the previous temporal point, then the temporal point is marked as duplicate data. The complete data of the first temporal point is retained and a binary identifier is stored to replace subsequent duplicate temporal points. Otherwise, the complete data of the temporal point is maintained. For associated data values not marked as duplicate data, the difference between them and the corresponding dimension associated data value of the previous valid temporal point is calculated as a difference result. The difference result is adaptively stored according to the numerical range of the difference result. The final compressed data packet is generated according to the coding result of the temporal skeleton and the compression result after adaptive storage. This application combines time-series hierarchical coding with differential compression and repetition marking, transforming the function of time-series coding from conventional data representation to a time-series framework that supports the effective operation of data value compression algorithms. The time-series skeleton provides predictability constraints on the sampling interval for differential compression, and repetition marking increases the probability of the differential result being zero. The three work together to improve compression efficiency. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the multimodal fusion characterization process for the evaluation of the platform area in this application;
[0045] Figure 2 This is a schematic diagram of the multimodal fusion characterization device for the evaluation of the platform area in this application. Detailed Implementation
[0046] Exemplary embodiments of the present disclosure will now be provided in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is to be understood that various forms of implementation of the present disclosure are intended and should not be limited to the embodiments set forth herein. Rather, the embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0047] This application provides a method for compressing historical power system data, which is applied in the field of power system data processing technology and is used to solve the problems of accuracy loss, low compression ratio or insufficient compression efficiency in the compression of historical power system data in the prior art.
[0048] like Figure 1 As shown, the method for compressing historical power system data includes:
[0049] S101. Obtain a set of historical power system data from the power system data acquisition terminal. The set of historical data includes time sequence identification information and multi-dimensional related data values.
[0050] The time sequence identifier information represents the data acquisition time, accurate to the millisecond level. The format of the time sequence identifier information is uniformly converted to a Unix timestamp, in milliseconds, denoted as... , The time series number, .
[0051] The multi-dimensional associated data values include power equipment operating parameters, electricity metering data, and power grid status data. Power equipment operating parameters include voltage, current, power, and temperature; electricity metering data includes electricity consumption and power generation; and power grid status data includes frequency and phase. Each dimension of data is uniformly converted to a 64-bit floating-point number, denoted as... , For data dimension index, .
[0052] Record the total number of time series points N and the number of data dimensions M to provide a reference for subsequent sliding window statistics and dictionary capacity settings.
[0053] Historical data sets can be obtained from power system data acquisition terminals, databases, or cloud storage.
[0054] S102. Perform abnormal data detection on the historical data set and remove abnormal data.
[0055] A dual assessment is conducted using the mean-standard deviation criterion and threshold values specific to the power industry data. This applies to each data dimension. Calculate the mean of all data in this dimension. and standard deviation The formula is:
[0056]
[0057]
[0058] in, For the first The mean of each data dimension; For the first Standard deviation of each data dimension; This represents the total number of time series data points. The time series index is the number of the time series points, and its value range is [value range missing]. to ; For the first The time point, the Related data values for each data dimension; This refers to the data dimension number.
[0059] like If so, it is preliminarily determined to be abnormal data.
[0060] At the same time, hard thresholds are set in accordance with power industry standards. If the data is initially judged to be normal but exceeds the industry threshold, it is still judged as abnormal data.
[0061] Industry thresholds include voltage dimension thresholds set to... The current dimension threshold is set to .
[0062] The complete time series points corresponding to the abnormal data are directly removed, and the abnormal time series point number and reason are recorded to generate an "Abnormal Data Report".
[0063] S103. Complete the missing data in the historical data set after removing abnormal data.
[0064] Iterate through the data of all time series points in each dimension. If any dimension of a time series point has empty or invalid data, it is determined to be a missing data point.
[0065] If data points are missing The previous valid timing point and the next valid time series point All exist Then, linear interpolation is used to complete the equation, and the formula is:
[0066]
[0067] in, For the first The missing data point, the first The completed data values for each data dimension; For the first Before the missing data point, the first The first valid time series point, the first Data values for each data dimension; For the first After the missing data point, the th The first valid time series point, the first Data values for each data dimension; The time series index of the missing data point; The interval between the missing data point and the previous valid time series point is the number of intervals. ; The interval between the missing data point and the next valid time series point is the number of intervals. ; This refers to the data dimension number.
[0068] If the missing data point is the first time series point If the missing data point is the last valid time series point, then the missing data point will be filled with the data from the next valid time series point; if the missing data point is the last valid time series point, then the missing data point will be filled with the data from the next valid time series point. If the number of consecutive missing time series points exceeds 5, an alarm will be triggered, and the user can choose whether to fill the missing data with the average of historical data from the same period or discard the missing data.
[0069] S104. Sort the completed historical data set according to the order of the time sequence identifier information, and select the time sequence identifier of the first time sequence point after sorting as the reference time sequence point.
[0070] All valid time series points after processing are sorted in ascending order according to their time series identifiers to ensure data temporal continuity. The time series point numbers are then renumbered. , This represents the number of valid time series points after preprocessing. The first time series point is selected after sorting. The timing identifier information is used as the reference timing point. .
[0071] S105. Based on the reference time series point, calculate the time span difference between each sorted time series point and the reference time series point.
[0072] For the sorted... Time series points Calculate its relationship with the reference time point Time span difference The formula is:
[0073]
[0074] in, For the first The time span difference between each time series point and the baseline time series point; For the first Unix timestamps for each time point; The Unix timestamp as the baseline time point; The time series number and .
[0075] S106. Based on the numerical range of the time span difference, the time span difference is divided into multiple coding levels, and the time span difference is binary encoded according to the coding level to construct a timing skeleton.
[0076] Based on the statistical patterns of power system data acquisition scenarios, four coding levels are defined.
[0077] The numerical range, identifier bits, and data bit lengths for each level are as follows:
[0078] The time span difference range for the first level is: Milliseconds, with a flag of "0", a data bit length of 7 bits, and a total encoding length of 8 bits, suitable for high-frequency acquisition scenarios.
[0079] The time span difference range for the second level is: The milliseconds value is marked with a "10" flag, the data bit length is 12 bits, and the total encoding length is 14 bits, making it suitable for conventional data acquisition scenarios.
[0080] The time span difference range for the third level is: The millisecond value is marked as "110", the data bit length is 16 bits, and the total encoding length is 19 bits, making it suitable for low-frequency acquisition scenarios.
[0081] The fourth level has a time span difference range greater than 65535 milliseconds, an identifier of "1111", a data bit length of 32 bits, and a total encoding length of 36 bits, making it suitable for ultra-long time span acquisition scenarios.
[0082] During encoding, for each Determine its encoding level and perform binary encoding according to the rules of the corresponding level. For example, if If it belongs to the first level, the code is "0" + "0110010", and the final code result is "00110010"; if If it belongs to the second level, the code is "10" + "000001100100", and the final code result is "10000001100100"; if If it belongs to the third level, the code is "110" + "0010011100010000", and the final code result is "1100010011100010000"; if If it belongs to the fourth level, the code is “1111” + “000000000000000011000011010100000”, and the final code result is “111100000000000000011000011010100000”.
[0083] Encoding and storage optimization involves concatenating the encoding results of all timing points in chronological order to form a continuous binary data stream. A 1-byte "hierarchical encoding identifier" is added to the beginning of the data stream. This identifier is fixed at 0x01 and is used to identify the encoding method of the timing identifier during subsequent decompression.
[0084] After encoding, the encoded binary data stream is checked using CRC32. The checksum is calculated and stored at the end of the data stream for verifying data integrity during decompression.
[0085] S107. For each time point in the time sequence skeleton, if the time interval between it and the time point at the previous moment falls within the time span range of the corresponding coding level in the time sequence skeleton, and the associated data value of the time point is completely consistent with the associated data value of the time point at the previous moment, then the time point is marked as duplicate data, the complete data of the first time point is retained and a binary identifier is stored to replace the subsequent duplicate time points; otherwise, the complete data of the time point is maintained.
[0086] The repeated data refers to data segments where all consecutive time points satisfy the condition that the associated data values are completely identical and the time intervals all fall within the time span range of the corresponding encoding level. The default number of consecutive data points is... It also supports user-defined adjustments.
[0087] After being marked as duplicate data, the processing method is to retain the complete data of the first time point of the duplicate data segment, and store only a 1-bit binary identifier "0" for subsequent duplicate time points to replace the complete data, so as to reduce redundant storage.
[0088] The criteria for determining duplicate data also include that the time interval of the time sequence identification information must meet the preset collection frequency.
[0089] S108. For associated data values that are not marked as duplicate data, first construct and match a dynamic dictionary. If the match fails, calculate the difference between the associated data value of the corresponding dimension of the previous valid time series point as the difference result.
[0090] The initial construction of the dynamic dictionary library involves importing standard benchmark data of the power system and historical typical data of the target equipment.
[0091] Standard reference data for power systems includes industry-standard parameters such as rated voltage, rated current, standard frequency, and rated power factor. Historical typical data for the target equipment is extracted from the historical operating data of that power equipment, focusing on high-frequency data values from the past three months.
[0092] The initial dictionary entry contains "data identifier (unique ID) - data content (multi-dimensional data combination or single-dimensional data value) - frequency of occurrence". The initial data identifier is a 16-bit unsigned integer, ranging from 0 to 65535, and the frequency of occurrence is initialized to 0.
[0093] The dynamic dictionary database is updated by including sliding window statistics and the identification and addition of high-frequency data.
[0094] Among them, sliding window statistics are achieved by setting the sliding window size to... The Users can adjust the frequency statistics of preprocessed correlated data values in windows, based on the amount of data, including single-dimensional data values. The frequency of occurrence of combinations with multi-dimensional data.
[0095] High-frequency data determination involves setting a frequency threshold. If the frequency of occurrence of a certain data value or data combination If an entry is not present in the dynamic dictionary, it is added as a new entry, an unused data identifier is assigned, and its frequency is recorded as a statistical value within the current window; if it already exists in the dictionary, its frequency is updated.
[0096] Dictionary capacity control is setting the maximum capacity of the dictionary database. When the number of dictionary entries reaches When the time comes, the elimination mechanism is triggered, removing the entry with the lowest frequency. If there are entries with the same frequency, the entry that was not most recently hit is removed first.
[0097] The dictionary database needs to be updated, including new entries, removed entries, and timestamps, to support dictionary database backtracking and recovery.
[0098] Multi-dimensional correlation data values for each time series point Prioritize multi-dimensional data combination matching, and combine all dimensions of data at the current time point as a whole to query the dynamic dictionary.
[0099] If a match is found, the corresponding data identifier is stored directly.
[0100] If a match fails, attempt to match data values in a single dimension, for each dimension. of The dictionary is queried separately. If a data value for a certain dimension matches the dictionary, the "dictionary identifier (1 bit '1') + data identifier (16 bits)" for that dimension is stored. If a data value for a certain dimension does not match the dictionary, the "non-dictionary identifier (1 bit '0')" for that dimension is stored, and the difference operation process is initiated.
[0101] Difference operations are performed on dimension data values that are not found in the dictionary. Calculate the dimensional data value corresponding to the previous time series point. Difference results The formula is:
[0102]
[0103] in, For the first The time point and the The time point, the The difference results of related data values in each data dimension; The sign is for the difference. For the first The time point, the Related data values for each data dimension; For the first The time point, the Related data values for each data dimension; The time sequence number; This refers to the data dimension number.
[0104] like For the first time series point, where there is no preceding data for comparison, the original 64-bit binary value of that data is directly stored as the differential reference value. Differential result. It supports both positive and negative values, represented in two's complement form.
[0105] S109. Based on the numerical range of the difference results, adaptively store the difference results.
[0106] Among them, according to The numerical range is used with different storage lengths, and the specific rules are as follows:
[0107] like If the storage flag is "0", the differential value storage length is 0 bits, and the total storage length is 1 bit, it means that the data has no fluctuation and only the flag is needed.
[0108] like If the storage flag is "10", the differential value storage length is 7 bits, and the total storage length is 8 bits, it is suitable for small fluctuations.
[0109] like If the storage identifier is "110", the differential value storage length is 9 bits, and the total storage length is 11 bits, it is suitable for medium fluctuations.
[0110] like If the storage identifier is "1110", the differential value storage length is 12 bits, and the total storage length is 14 bits, which is suitable for larger fluctuations.
[0111] For other ranges, the storage flag is "1111", the differential value storage length is 32 bits, and the total storage length is 36 bits, which is suitable for large fluctuations.
[0112] For example, if The value corresponding to the previous time point ,but ,belong The range is stored as "10" plus "00000010", with a total length of 8 bits. The compressed results of the correlation data values of all time series points are concatenated in the order of "time series point number - dimension number" to form a compressed correlation data stream.
[0113] Add dictionary metadata to the header of the associated compressed data stream, including dictionary version (2 bytes), current number of dictionary entries (2 bytes), and dictionary entry index table (each entry occupies 20 bytes, including data identifier, data length, and data content summary), for rebuilding the dynamic dictionary during decompression.
[0114] Calculate the CRC32 checksum of the associated compressed data stream, store it at the end of the data stream, and add a 1-byte "associated data compression identifier", which is fixed at 0x02 and is used to identify the compression method during decompression.
[0115] S110. Generate the final compressed data packet based on the encoding result of the temporal skeleton and the compression result after adaptive storage.
[0116] The timing identifier layered encoded data stream and the associated compressed data stream are concatenated in the order of "timing identifier stream - associated data stream". A 1-byte "compression format total identifier", which is fixed at 0x00, and a 4-byte "total data length" are added to the header of the overall data stream. The "total data length" records the number of bytes in the overall compressed data stream, thus forming the final compressed data packet.
[0117] Simultaneously, statistics on compression effectiveness are calculated, including the data volume before and after compression, and the compression ratio is calculated using the following formula:
[0118]
[0119] Compression performance statistics are stored at the end of the compressed data packet to help users evaluate the compression effect.
[0120] This application also provides a multimodal fusion characterization device for transformer area assessment, comprising:
[0121] The acquisition module obtains a set of historical power system data from the power system data acquisition terminal. The set of historical data includes time sequence identification information and associated data values.
[0122] The correction module performs anomaly detection on the historical data set, removes anomaly data, and completes the missing data in the historical data set after removing anomaly data.
[0123] The sorting module sorts the completed historical data set according to the chronological order of the time sequence identifier information, and selects the time sequence identifier of the first time sequence point after sorting as the reference time sequence point.
[0124] The time difference module calculates the time span difference between each sorted time series point and the reference time series point based on the reference time series point;
[0125] The skeleton module divides the time span difference into multiple coding levels according to the numerical range of the time span difference, and performs binary encoding on the time span difference according to the coding level to construct a time-series skeleton.
[0126] The judgment module determines that for each time point in the time sequence skeleton, if the time interval between it and the previous time point falls within the time span range of the corresponding coding level in the time sequence skeleton, and the associated data value of the time point is completely consistent with the associated data value of the previous time point, then the time point is marked as duplicate data, the complete data of the first time point is retained and a binary identifier is stored to replace the subsequent duplicate time points; otherwise, the complete data of the time point is maintained.
[0127] The results module calculates the difference between the associated data values that are not marked as duplicate data and the corresponding dimension associated data values of the previous valid time series point as the difference result; adaptively stores the difference result according to the numerical range of the difference result; and generates the final compressed data packet according to the encoding result of the time series skeleton and the compression result after adaptive storage.
[0128] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0129] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the above-described method.
[0130] The above embodiments are provided to enable those skilled in the art to understand and apply this application. Those skilled in the art will readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without inventive effort. Therefore, this application is not limited to the above embodiments, and any improvements and modifications made to this application based on the disclosure thereof should be within the scope of protection of this application.
Claims
1. A method for compressing historical data of a power system, characterized in that, include: The historical data set of the power system is obtained from the power system data acquisition terminal. The historical data set includes time sequence identification information and associated data values. Anomaly detection is performed on the historical data set, and abnormal data is removed; Complete the missing data in the historical dataset after removing outliers; The completed historical data set is sorted according to the chronological order of the time sequence identifier information, and the time sequence identifier of the first time sequence point after sorting is selected as the reference time sequence point. Based on the baseline time series point, calculate the time span difference between each sorted time series point and the baseline time series point; Based on the numerical range of the time span difference, the time span difference is divided into multiple coding levels, and the time span difference is binary encoded according to the coding level to construct a time sequence skeleton; For each time point in the time sequence skeleton, if the time interval between it and the previous time point falls within the time span range of the corresponding coding level in the time sequence skeleton, and the associated data value of the time point is completely consistent with the associated data value of the previous time point, then the time point is marked as duplicate data, the complete data of the first time point is retained and a binary identifier is stored to replace the subsequent duplicate time points; otherwise, the complete data of the time point is maintained. For associated data values that are not marked as duplicates, the difference between them and the corresponding dimension associated data values of the previous valid time series point is calculated as the difference result. Based on the numerical range of the difference results, the difference results are adaptively stored; Based on the encoding result of the temporal skeleton and the compression result after adaptive storage, the final compressed data packet is generated.
2. The method according to claim 1, characterized in that, Anomaly detection and removal are performed on the historical data set, including: The system employs a dual criterion of mean and standard deviation and an industry threshold. If the value of the associated data at any time point exceeds the sum of the mean and three times the standard deviation of the associated data or is lower than the difference between the mean and three times the standard deviation, and also exceeds the industry threshold, then the data at any time point is determined to be abnormal and is removed. Otherwise, maintain the complete data for any of the stated time points.
3. The method according to claim 1, characterized in that, Complete missing data in the historical dataset after removing outliers, including: If there are valid time series points before and after the missing data point, then linear interpolation is used to complete the missing data point by using the associated data values of the valid time series points before and after the missing data point. If the missing data point is the first or last data point, then the associated data value of the adjacent valid time series point is used for one-way filling; An alarm is triggered if the number of consecutive missing data points exceeds the preset alarm threshold.
4. The method according to claim 1, characterized in that, Based on the numerical range of the time span difference, the time span difference is divided into multiple coding levels. The time span difference is then binary-encoded according to these coding levels to construct a time-series skeleton, including: The multiple encoding levels include four levels: the first level has a time span difference range of 0 to 127 milliseconds and uses 8-bit encoding; the second level has a time span difference range of 128 to 4095 milliseconds and uses 14-bit encoding; the third level has a time span difference range of 4096 to 65535 milliseconds and uses 19-bit encoding; and the fourth level has a time span difference range greater than 65535 milliseconds and uses 36-bit encoding.
5. The method according to claim 1, characterized in that, The repeated data refers to a data segment in which a predetermined number of consecutive time points all have completely identical associated data values and the time intervals all fall within the time span range of the corresponding encoding level.
6. The method according to claim 1, characterized in that, For associated data values not marked as duplicates, the difference between them and the corresponding dimension associated data value of the previous valid time series point is calculated as the difference result, including: Before calculating the difference result, a dynamic dictionary is constructed. The initial entries of the dynamic dictionary include industry standard parameters and typical historical data of the equipment. A unique data identifier is assigned to each entry in the dynamic dictionary. For associated data values that are not marked as duplicate data, dictionary matching is first performed according to multi-dimensional combinations. If a match is successful, the corresponding data identifier is stored. If the multi-dimensional combination matching fails, dictionary matching is performed one by one according to a single dimension. If a match is successful, the corresponding data identifier is stored. If no matching is successful for any dimension, the difference between the associated data value and the corresponding dimension of the previous valid time series point is calculated as the difference result.
7. The method according to claim 6, characterized in that, The construction of the dynamic dictionary library includes: A sliding window is used to count the frequency of occurrence of the associated data value. If the frequency of occurrence of the associated data value is greater than or equal to a preset frequency threshold and is not in the dynamic dictionary, the associated data value is added to the dynamic dictionary and a data identifier is assigned. If the number of entries in the dynamic dictionary reaches the preset capacity limit, the entry with the lowest frequency of occurrence will be discarded.
8. A compression device for historical power system data, characterized in that, include: The acquisition module acquires a set of historical power system data from the power system data acquisition terminal. The set of historical data includes time sequence identification information and associated data values. The removal module performs anomaly detection on the historical data set and removes abnormal data. The data completion module completes missing data in the historical dataset after removing outlier data. The sorting module sorts the completed historical data set according to the chronological order of the time sequence identifier information, and selects the time sequence identifier of the first time sequence point after sorting as the reference time sequence point. The difference module calculates the time span difference between each sorted time series point and the benchmark time series point based on the benchmark time series point. The skeleton module divides the time span difference into multiple coding levels according to the numerical range of the time span difference, and performs binary encoding on the time span difference according to the coding level to construct a time-series skeleton. The judgment module determines that for each time point in the time sequence skeleton, if the time interval between it and the previous time point falls within the time span range of the corresponding coding level in the time sequence skeleton, and the associated data value of the time point is completely consistent with the associated data value of the previous time point, then the time point is marked as duplicate data, the complete data of the first time point is retained and a binary identifier is stored to replace the subsequent duplicate time points; otherwise, the complete data of the time point is maintained. The results module calculates the difference between the associated data values that are not marked as duplicate data and the corresponding dimension associated data values of the previous valid time series point as the difference result. The storage module adaptively stores the difference results according to the numerical range of the difference results; The data module generates the final compressed data packet based on the encoding result of the temporal skeleton and the compression result after adaptive storage.
9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any one of the methods described in claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform any one of the methods described in claims 1 to 7.