Data compression method and device, equipment and storage medium

By dividing segments according to data characteristics and adopting different compression methods, using sliding windows and improved algorithms, the problem that data compression methods in the prior art are difficult to balance compression accuracy and efficiency, and efficient and accurate data compression is achieved.

CN119093942APending Publication Date: 2024-12-06STATE GRID TIANJIN ELECTRIC POWER COMPANY +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411367523.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing data compression methods are difficult to balance compression accuracy and compression efficiency, and cannot ensure data compression efficiency while taking into account data compression accuracy.

Method used

According to the characteristics of the data, the data is divided into multiple preset types of segments, and a different compression method is adopted for each segment, including determining the segment type based on the fluctuation degree and time length of the sliding window, and using improved fuzzy transformation Algorithms and SAX algorithms are compressed.

Benefits of technology

It realizes the data compression accuracy while ensuring data compression efficiency, improves the compression effect and compression speed, and solves the problem of difficult to balance compression accuracy and efficiency in the prior art.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119093942A_ABST
    Figure CN119093942A_ABST
Patent Text Reader

Abstract

The invention provides a data compression method and device, equipment and a storage medium, and the method comprises the steps: dividing data into a plurality of preset types of sections according to the characteristics of the data; a target compression mode corresponding to a to-be-compressed target section is obtained, and the target compression mode comprises a preset compression mode corresponding to the type of the target section; and compressing the target section according to the target compression mode, dividing data into different types of sections according to data characteristics, presetting a proper compression mode for the data characteristics of each type of section, and calling the compression mode corresponding to the section type to compress the sections. The technical problem that a data compression mode in the prior art is difficult to balance the compression precision and the compression efficiency is solved, and the technical effect that the data compression precision is considered while the data compression efficiency is guaranteed is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a data compression method, device, equipment and storage medium. Background Art

[0002] Current data compression methods include lossy compression and lossless compression. Lossless compression achieves the purpose of data compression by removing redundant information in some original representation information and reducing the redundancy in the information. Lossless compression achieves the purpose of data compression by removing redundant information in some original representation information and reducing the redundancy in the information. This method can retain all the information of the original data, but the compression ratio is lower than that of lossy compression. Lossy compression achieves the purpose of data compression by identifying unnecessary information in the data and removing it. This method will cause the original information to be lost within a certain range.

[0003] However, although the existing lossless compression has a high compression accuracy, the compression ratio is low, while the lossy compression has a high compression ratio but a low compression accuracy. It can be seen that the existing data compression method is still difficult to balance the reconstruction accuracy and compression efficiency in the compression process. Summary of the invention

[0004] The present invention provides a data compression method, device, equipment and storage medium to solve the technical problem that the data compression method in the prior art is difficult to balance the compression accuracy and compression efficiency.

[0005] In a first aspect, the present invention provides a data compression method, comprising:

[0006] Divide the data into multiple preset types of segments according to the characteristics of the data;

[0007] Obtaining a target compression mode corresponding to the target segment to be compressed, wherein the target compression mode includes a preset compression mode corresponding to the type of the target segment;

[0008] Compress the target segment according to the target compression method.

[0009] Furthermore, the types of segments include non-event segments, long event segments, and short event segments.

[0010] Furthermore, according to the characteristics of the data, the data is divided into multiple preset types of segments, including:

[0011] Based on a sliding window with a preset number of data bits, determine the degree of fluctuation of data within the sliding window;

[0012] When the fluctuation degree meets the preset fluctuation threshold, the segment in the sliding window is determined as an event segment;

[0013] When the time lengths corresponding to two adjacent event segments are less than the preset time, the two adjacent event segments are merged into a new event segment;

[0014] Repeat the step of merging two adjacent event segments into a new event segment when the sum of the time lengths corresponding to the two adjacent event segments is less than the preset time, until the sum of the time lengths corresponding to all adjacent time periods and / or the new time period is greater than or equal to the preset time, and take all event segments and the new event segment as the target event segment;

[0015] When the fluctuation degree does not meet the preset fluctuation threshold, the segment in the sliding window is determined as a non-event segment;

[0016] According to the corresponding time length of the target event segment, the target event segment is divided into a long event segment and a short event segment.

[0017] Furthermore, according to the characteristics of the data, the data is divided into multiple preset types of segments, including:

[0018] Obtain a first preset formula, wherein the first preset formula satisfies:

[0019] ΔP=mean(W i POST )-mean(W i PRE ),

[0020] D(i)=[iw,i+w],

[0021] W i POST = {P i+1 ,P i+2 ,...,P i+w},

[0022] W i PRE = {P i-w ,P i-w+1 ,...,P i-1},

[0023] ΔT=ji-2w,

[0024] ΔP is the fluctuation detection variable, ΔT is the time detection variable, D(i) is the sliding window, w is half of the preset length value, W i POST = {P i+1 ,P i+2 ,...,P i+w} is the data of the second half of the sliding window, W i PRE = {P i-w ,P i-w+1,...,P i-1} is the data of the first half of the sliding window, mean(·) is the function for finding the mean, i is the serial number of the first sliding window in the segment, and j is the serial number of the last sliding window in the segment;

[0025] When the fluctuation detection variable is less than or equal to a preset fluctuation threshold, the segment in the sliding window is determined to be a non-event segment;

[0026] When the fluctuation detection variable is greater than a preset fluctuation threshold, the segment is determined to be an event segment;

[0027] When the time detection variable is less than or equal to the preset time value, the segment is determined to be a short event segment;

[0028] When the time detection variable is greater than a preset time value, the segment is determined to be a long event segment.

[0029] Furthermore, obtaining a target compression mode corresponding to the target segment to be compressed includes:

[0030] When the target segment is a long event segment, the long event segment is compressed based on the improved fuzzy transformation algorithm;

[0031] When the target segment is a short event segment, the short event segment is not compressed;

[0032] When the target segment is a non-event segment, the non-event segment is compressed based on the SAX algorithm.

[0033] Furthermore, the long event segment is compressed based on the improved fuzzy transformation algorithm, including:

[0034] According to the long event segment, generate the data matrix D∈N×M;

[0035] Based on the second preset formula, the data matrix is ​​fuzzy transformed to obtain the target matrix, wherein the second preset formula satisfies:

[0036]

[0037] P=A T DB,

[0038]

[0039]

[0040] F kl ∈n×m is the target matrix, P kl is the data matrix, Q kl For fuzzy partitioning, is the N×M identity matrix, A={A 1 ,...,An} is the first fuzzy partition for the interval [1, N], B = {B 1 ,..., B m} is the second fuzzy partition for the interval [1, M]. The fuzzy partition includes the first fuzzy partition and the second fuzzy partition. n < N, m < M, k ∈ (1, n), l ∈ (1, m), k is an integer, l is an integer, a is a positive integer less than the number of the first fuzzy partition, and b is a positive integer less than the number of the second fuzzy partition.

[0041] In a second aspect, the present invention provides a data compression device, including:

[0042] A partitioning module, configured to partition data into multiple sections of preset types according to the characteristics of the data;

[0043] An obtaining module, configured to obtain a target compression method corresponding to a target section to be compressed, where the target compression method includes a preset compression method corresponding to the type of the target section;

[0044] A compression module, configured to compress the target section according to the target compression method.

[0045] Further, the partitioning module includes a first determination unit, a second determination unit, a first merging unit, a second merging unit, a third determination unit, and a partitioning unit;

[0046] The first determination unit is configured to determine the degree of fluctuation of the data within a sliding window based on a sliding window of a preset data bit number;

[0047] The second determination unit is configured to determine the section within the sliding window as an event section when the degree of fluctuation meets a preset fluctuation threshold;

[0048] The first merging unit is configured to merge two adjacent event sections into a new event section when the time lengths corresponding to two adjacent event sections are less than a preset time;

[0049] The second merging unit is configured to repeat the step of merging two adjacent event sections into a new event section when the sum of the time lengths corresponding to two adjacent event sections is less than a preset time until the sum of the time lengths corresponding to all adjacent time periods and / or new time periods is greater than or equal to the preset time, and use all event sections and new event sections as target event sections;

[0050] The third determination unit is configured to determine the section within the sliding window as a non-event section when the degree of fluctuation does not meet the preset fluctuation threshold;

[0051] The partitioning unit is configured to divide the target event section into a long event section and a short event section according to the time length corresponding to the target event section.

[0052] Further, the division module includes an acquisition unit, a fourth determination unit, a fifth determination unit, a sixth determination unit, and a seventh determination unit;

[0053] An acquiring unit is used to acquire a first preset formula, wherein the first preset formula satisfies:

[0054] ΔP=mean(W i POST )-mean(W i PRE ),

[0055] D(i)=[iw,i+w],

[0056] W i POST = {P i+1 ,P i+2 ,...,P i+w},

[0057] W i PRE = {P i-w ,P i-w+1 ,...,P i-1},

[0058] ΔT=ji-2w,

[0059] ΔP is the fluctuation detection variable, ΔT is the time detection variable, D(i) is the sliding window, w is half of the preset length value, W i POST = {P i+1 ,P i+2 ,...,P i+w} is the data of the second half of the sliding window, W i PRE = {P i-w ,P i-w+1 ,...,P i-1} is the data of the first half of the sliding window, mean(·) is the function for finding the mean, i is the serial number of the first sliding window in the segment, and j is the serial number of the last sliding window in the segment;

[0060] A fourth determining unit, configured to determine that a segment in the sliding window is a non-event segment when the fluctuation detection variable is less than or equal to a preset fluctuation threshold;

[0061] A fifth determining unit, configured to determine the segment as an event segment when the fluctuation detection variable is greater than a preset fluctuation threshold;

[0062] a sixth determining unit, configured to determine that the segment is a short event segment when the time detection variable is less than or equal to a preset time value;

[0063] A seventh determination unit, configured to determine that the section is a long event section when the time detection variable is greater than a preset time value.

[0064] Further, the acquisition module includes a first compression unit, a second compression unit, and a third compression unit;

[0065] The first compression unit is configured to, when the target section is a long event section, compress the long event section based on an improved fuzzy transformation algorithm;

[0066] The second compression unit is configured to, when the target section is a short event section, not compress the short event section;

[0067] The third compression unit is configured to, when the target section is a non-event section, compress the non-event section based on the SAX algorithm.

[0068] Further, the first compression unit includes a generation unit and a transformation unit;

[0069] The generation unit is configured to generate a data matrix D∈N×M according to the long event section;

[0070] The transformation unit is configured to perform a fuzzy transformation on the data matrix based on a second preset formula to obtain a target matrix, where the second preset formula satisfies:

[0071]

[0072] P = A T DB,

[0073]

[0074]

[0075] F kl ∈n×m is the target matrix, P kl is the data matrix, Q kl is a fuzzy partition, is an identity matrix of N×M, A = {A 1 ,..., A n} is a first fuzzy partition of the interval [1, N], B = {B 1 ,..., B m} is a second fuzzy partition of the interval [1, M], the fuzzy partition includes the first fuzzy partition and the second fuzzy partition, n < N, m < M, k∈(1, n), l∈(1, m), k is an integer, l is an integer, a is a positive integer less than the number of the first fuzzy partition, and b is a positive integer less than the number of the second fuzzy partition.

[0076] In a third aspect, the present invention provides an electronic device, including:

[0077] one or more processors;

[0078] A memory for storing one or more programs;

[0079] When the one or more programs are executed by the one or more processors, the one or more processors implement the data compression method as described in the first aspect.

[0080] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data compression method as described in the first aspect.

[0081] The embodiment of the present invention divides data into multiple different types of segments according to the characteristics of the data, pre-sets a suitable compression method according to the data characteristics of each type of segment, and calls the compression method corresponding to the segment type to compress the segment. By flexibly determining the method for compressing data according to the data characteristics, it achieves a balance between compression effect and compression speed, solves the technical problem that it is difficult to balance compression accuracy and compression efficiency in the data compression method in the prior art, and achieves the technical effect of balancing data compression accuracy while ensuring data compression efficiency.

[0082] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 A schematic diagram of a data compression method provided by an embodiment of the present invention;

[0085] Figure 2 A schematic diagram of a flow chart of an optional data compression method provided by an embodiment of the present invention;

[0086] Figure 3 A schematic diagram of result indicators of data compression according to sliding windows with different numbers of data bits provided in an embodiment of the present invention;

[0087] Figure 4A comparison chart of CR and RMSE indicators of data compression results of the data compression method provided by an embodiment of the present invention and the FLDC algorithm;

[0088] Figure 5 A comparison chart of CR and MPPE indicators of data compression results of the data compression method provided by an embodiment of the present invention and the FLDC algorithm;

[0089] Figure 6 A schematic diagram of the structure of a data compression device provided by an embodiment of the present invention;

[0090] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0093] In order to solve the technical problem that it is difficult to balance compression accuracy and compression efficiency in the data compression method in the prior art, an embodiment of the present invention provides a data compression method, such as Figure 1 As shown: Figure 1 A flow chart of a data compression method provided by an embodiment of the present invention, the method comprising:

[0094] S101: Divide the data into multiple preset types of segments according to the characteristics of the data;

[0095] The characteristic may refer to the degree of fluctuation of the data, which may be the difference between the maximum and minimum values ​​of the data within a preset number of digits, or the difference between the mean values ​​within adjacent preset number of digits.

[0096] Specifically, according to the fluctuation degree of the data, the data is divided into a plurality of segments of different preset types. The segments may be data of preset digits, and the types may be preset according to the data characteristics.

[0097] S102: Obtaining a target compression mode corresponding to the target segment to be compressed, wherein the target compression mode includes a preset compression mode corresponding to the type of the target segment;

[0098] Specifically, the current segment to be compressed is used as the target segment, and the preset compression method corresponding to the target segment is called, so that the data can be compressed in real time, thereby ensuring the compression efficiency.

[0099] S103: Compress the target segment according to the target compression method.

[0100] Specifically, the target segment is compressed according to the obtained compression method corresponding to the type of the target segment.

[0101] The embodiment of the present invention divides data into multiple different types of segments according to the characteristics of the data, pre-sets a suitable compression method according to the data characteristics of each type of segment, and calls the compression method corresponding to the segment type to compress the segment. By flexibly determining the method for compressing data according to the data characteristics, it achieves a balance between compression effect and compression speed, solves the technical problem that it is difficult to balance compression accuracy and compression efficiency in the data compression method in the prior art, and achieves the technical effect of balancing data compression accuracy while ensuring data compression efficiency.

[0102] In an optional embodiment, the types of segments include non-event segments, long event segments, and short event segments.

[0103] Specifically, the segments can be pre-set to include non-event segments, long event segments and short event segments, the data bits include data segments with a preset number of bits, the segments are divided into data segments and non-data segments according to the degree of fluctuation of the data in the segments with the preset number of bits, and the data are divided into long event segments and short data segments according to the time corresponding to the data in the segments with the preset number of bits.

[0104] Exemplarily, the present invention can be used for scenarios of load power consumption. The event segment in this scenario is the process of a change in the operating state of an electrical appliance, including the switching of electrical equipment on and off and between operating state modes. This process is reflected in power as a change in the power value. The waveform is relatively complex and contains more valuable information, supporting related data analysis and applications. The non-event segment in this scenario is the remaining data segment between two event segments. The operating state of this segment does not change and is relatively stable, without obvious fluctuations in the power value. Corresponding data compression processing is performed on this segment, with higher compression efficiency.

[0105] It should be noted that with the continuous development of modern power systems, advanced measurement systems are becoming more and more popular, the level of grid informatization is constantly improving, and smart measurement meters and other devices have generated a large amount of digital and intelligent measurement information. The implicit information has valuable use value. Using this information can provide better data analysis resources for user behavior habit analysis, load forecast estimation, operation status awareness monitoring and other fields, conduct further technical research and development, and improve the efficiency of information resource use. However, with the continuous deployment and popularization of smart measurement meters, the data generated is showing a trend of continuous increase. The generation of a large amount of data has also brought tremendous pressure to the data storage, transmission, analysis and use. In order to better use data resources and reduce the pressure of data storage and transmission, the embodiment of the present invention provides a data compression method applied to load power consumption data.

[0106] In an optional embodiment, the data is divided into multiple preset types of segments according to the characteristics of the data, including:

[0107] Based on a sliding window with a preset number of data bits, determine the degree of fluctuation of data within the sliding window;

[0108] When the fluctuation degree meets the preset fluctuation threshold, the segment in the sliding window is determined as an event segment;

[0109] When the time lengths corresponding to two adjacent event segments are less than the preset time, the two adjacent event segments are merged into a new event segment;

[0110] Repeat the step of merging two adjacent event segments into a new event segment when the sum of the time lengths corresponding to the two adjacent event segments is less than the preset time, until the sum of the time lengths corresponding to all adjacent time periods and / or the new time period is greater than or equal to the preset time, and take all event segments and the new event segment as the target event segment;

[0111] When the fluctuation degree does not meet the preset fluctuation threshold, the segment in the sliding window is determined as a non-event segment;

[0112] According to the corresponding time length of the target event segment, the target event segment is divided into a long event segment and a short event segment.

[0113] In an optional embodiment, the data is divided into multiple preset types of segments according to the characteristics of the data, including:

[0114] Obtain a first preset formula, wherein the first preset formula satisfies:

[0115] ΔP=mean(W i POST )-mean(W i PRE ),

[0116] D(i)=[iw,i+w],

[0117] W i POST = {P i+1 ,P i+2 ,...,P i+w},

[0118] W i PRE = {P i-w ,P i-w+1 ,...,P i-1},

[0119] ΔT=ji-2w,

[0120] ΔP is the fluctuation detection variable, ΔT is the time detection variable, D(i) is the sliding window, w is half of the preset length value, W i POST = {P i+1 ,P i+2 ,...,P i+w} is the data of the second half of the sliding window, W i PRE = {P i-w ,P i-w+1 ,...,P i-1} is the data of the first half of the sliding window, mean(·) is the function for finding the mean, i is the serial number of the first sliding window in the segment, and j is the serial number of the last sliding window in the segment;

[0121] When the fluctuation detection variable is less than or equal to a preset fluctuation threshold, the segment in the sliding window is determined to be a non-event segment;

[0122] When the fluctuation detection variable is greater than a preset fluctuation threshold, the segment is determined to be an event segment;

[0123] When the time detection variable is less than or equal to the preset time value, the segment is determined to be a short event segment;

[0124] When the time detection variable is greater than a preset time value, the segment is determined to be a long event segment.

[0125] Exemplarily, in an embodiment of the present invention, for a scenario of load power consumption, a sliding window is set to perform a difference operation on the power values ​​before and after, a threshold is set to determine the time period, and a segment that meets a preset fluctuation threshold is screened out as an event segment. Optionally, the remaining part except the event segment can be regarded as a non-event segment, and the segment that does not meet the preset fluctuation threshold can also be regarded as a non-event segment.

[0126] In an optional embodiment, obtaining a target compression mode corresponding to a target segment to be compressed includes:

[0127] When the target segment is a long event segment, the long event segment is compressed based on the improved fuzzy transformation algorithm;

[0128] When the target segment is a short event segment, the short event segment is not compressed;

[0129] When the target segment is a non-event segment, the non-event segment is compressed based on the SAX algorithm.

[0130] It should be noted that the embodiment of the present invention divides data into long event segments, short event segments and non-event segments. Due to the different data characteristics of different segments, different compression methods are pre-set for different segments. For short event segments, due to the short data volume and low compression efficiency, the original value transmission and storage method is adopted; while for long event segments, there are more data points, richer information, and more obvious waveform changes, so fuzzy transformation in a rectangular domain is used for data compression; the non-event segment has a large amount of data, small fluctuations, and is relatively stable, with a high compression efficiency. The SAX algorithm is used for processing and data compression. After storage and transmission, the data is finally reconstructed. While ensuring a certain reconstruction accuracy, the degree of compression can be increased to achieve a better comprehensive compression effect.

[0131] In an optional embodiment, compressing the long event segment based on an improved fuzzy transformation algorithm includes:

[0132] According to the long event segment, generate the data matrix D∈N×M;

[0133] Based on the second preset formula, the data matrix is ​​fuzzy transformed to obtain the target matrix, wherein the second preset formula satisfies:

[0134]

[0135] P=A T DB,

[0136]

[0137] F kl ∈n×m is the target matrix, P kl is the data matrix, Qkl is the fuzzy partition, is the N×M identity matrix, A={A 1 ,...,A n} is the first fuzzy partition of interval [1,N], B={B 1 ,...,B m} is the second fuzzy partition for the interval [1, M], and the fuzzy partition includes the first fuzzy partition and the second fuzzy partition. n < N, m < M, k ∈ (1, n), l ∈ (1, m), k is an integer, l is an integer, a is a positive integer less than the number of the first fuzzy partition, and b is a positive integer less than the number of the second fuzzy partition.

[0138] In an optional embodiment, when performing data compression on the non-event segment, specifically, first set the number of time partitions and the number of symbols. For the determination of these two variables, other variables need to be controlled. By varying and selecting a single variable, a suitable value that can both ensure a certain reconstruction accuracy and improve the compression degree is selected to perform data compression based on the SAX theory.

[0139] Exemplarily, in the embodiment of the present invention for the scenario of load power consumption, first use the PAA algorithm to perform dimensionality reduction processing on the collected load power curve. This algorithm can efficiently achieve data dimensionality reduction while retaining the overall trend of the original load curve. First, it is necessary to receive the number of partitions n of the load curve set by relevant personnel. For a data sequence M = {M 1 , M 2 , M 3 ,..., M N} corresponding to a certain time segment, the corresponding time period is evenly divided into n segments, and the mean value of each divided segment is used to replace the corresponding data to form a new load curve, achieving the purpose of data dimensionality reduction. The solution method for the mean value of each segment is shown in the third preset formula:

[0140]

[0141] where N is the total number of corresponding data values, n is the set number of time segments, and M j is the data value at the corresponding time point. After re-expressing the load data reduced in dimension based on the PAA algorithm, symbols are used to represent the corresponding load segments. The normalized time series conforms to the characteristics of a Gaussian distribution. Therefore, the symbol representation uses the Gaussian distribution quantiles as the segmentation points between different symbols. When the number of symbol representations is Q, the corresponding symbol segmentation points are the Gaussian distribution quantile sequences <β 1 , β 2 , β 3 ,..., β Q-1 >. It is divided into <-∞, β 1 >, <β 1 , β 2 >, <β 2 , β 3 >, …, <β Q-1,+∞>There are Q partition segments in total, which can be represented by symbols such as "A", "B", "C", ..., "Q", etc. The obtained data mean is judged to fall into the interval and symbolically represented, as shown in the fourth preset formula:

[0142]

[0143] M s It represents the symbolic representation result of the load data between time period i-1 to i. Symbol is the symbol preset by relevant personnel. β j is the jth quantile of the Gaussian distribution.

[0144] Based on the above embodiment, the present invention provides an optional embodiment, which provides a reconstruction method for compressed data of different types of segments: for the target matrix obtained by fuzzy transformation of the matrix data of the long event segment in the rectangular domain, the fuzzy inverse transformation D F =AFB T Reconstruct the original matrix data; for short events, the original values ​​are transmitted and / or stored. During reconstruction, only the corresponding data needs to be read without any other processing; for non-event segments, the SAX algorithm is used for data compression, which needs to be reconstructed segment by segment based on the data's mean, sign, and preset information representing the position of each time partition.

[0145] The embodiment of the present invention uses the REDD public data of American households to conduct experiments. When the data sampling frequency is 1 Hz, the daily electricity consumption data of multiple households are selected to test and analyze the data compression effect. The data compression is performed on multiple types of electrical equipment such as ovens, refrigerators, dishwashers, lighting equipment, spin dryers, microwave ovens, and electric heaters. The effect of data compression is usually judged based on the following two indicators: the compression ratio CR is used to evaluate the amount of data reduction, and the RMSE and MPPE are used to measure the reconstruction accuracy. CR is an indicator corresponding to the degree of compression. The greater the degree of data compression, the greater the corresponding indicator value; RMSE and MPPE are both evaluation indicators related to reconstruction accuracy. The more accurate the reconstruction degree, the smaller the corresponding indicator value. Figure 2 As shown, Figure 2 A flowchart of an optional data compression method provided by an embodiment of the present invention. Figure 3 As shown, Figure 3 The schematic diagram of the result index of data compression according to the sliding window with different data bit numbers provided by the embodiment of the present invention is as follows. 6 is selected as the time partition number of the long event segment, 5 is selected as the time partition number of the non-event segment, and the number of symbols is set to 8. The result index after compression according to the data compression method provided by the embodiment of the present invention is compared with the result index after compression by the FLDC algorithm, and the reconstructed data is also compared. Figure 4 and Figure 5 As shown, Figure 4 A comparison chart of the CR and RMSE indicators of the data compression results of the data compression method provided by the embodiment of the present invention and the FLDC algorithm, Figure 5 A comparison chart of the CR and MPPE indicators of the data compression results of the data compression method provided by the embodiment of the present invention and the FLDC algorithm. Figure 4 and Figure 5 It can be seen that the reconstruction accuracy of the compressed data of the data compression method provided by the embodiment of the present invention is higher than that of FLDC, and the RSME and MPPE index values ​​are at a lower level. The average reconstruction accuracy obtained by the RSME index is 99.78%, and the average reconstruction accuracy obtained by the MPPE index is 99.92%. On this basis, the compression degree is also high. The data compression ratio of the data compression method proposed in the embodiment of the present invention is 21.90 on average, which is better than the average compression ratio of 2.14 of the FLDC algorithm. In the FLDC method, GVE is used to divide the ground state and the excitation state, which easily divides the low-amplitude fluctuation segment into the ground state, and uses the same value for reconstruction. The data compression method proposed in the embodiment of the present invention can reasonably compress data in different states, while ensuring a certain reconstruction accuracy, and improve the compression degree as much as possible.

[0146] Based on the same concept, an embodiment of the present invention provides a data compression device, such as Figure 2 As shown: Figure 2 A schematic diagram of the structure of a data compression device provided by an embodiment of the present invention, the device comprising:

[0147] A division module 201 is used to divide the data into a plurality of preset types of segments according to the characteristics of the data;

[0148] An acquisition module 202 is used to acquire a target compression method corresponding to a target segment to be compressed, wherein the target compression method includes a preset compression method corresponding to a type of the target segment;

[0149] The compression module 203 is used to compress the target segment according to the target compression method.

[0150] The embodiment of the present invention divides data into multiple different types of segments according to the characteristics of the data, pre-sets a suitable compression method according to the data characteristics of each type of segment, and calls the compression method corresponding to the segment type to compress the segment. By flexibly determining the method for compressing data according to the data characteristics, it achieves a balance between compression effect and compression speed, solves the technical problem that it is difficult to balance compression accuracy and compression efficiency in the data compression method in the prior art, and achieves the technical effect of balancing data compression accuracy while ensuring data compression efficiency.

[0151] Further, the division module 201 includes a first determination unit, a second determination unit, a first merging unit, a second merging unit, a third determination unit and a division unit;

[0152] A first determining unit, configured to determine a degree of fluctuation of data in a sliding window based on a sliding window with a preset number of data bits;

[0153] A second determining unit, configured to determine a segment in the sliding window as an event segment when the fluctuation degree meets a preset fluctuation threshold;

[0154] A first merging unit, configured to merge two adjacent event segments into a new event segment when the time lengths corresponding to the two adjacent event segments are less than a preset time;

[0155] A second merging unit is used to repeat the step of merging two adjacent event segments into a new event segment when the sum of the time lengths corresponding to the two adjacent event segments is less than a preset time, until the sum of the time lengths corresponding to all adjacent time periods and / or the new time period is greater than or equal to the preset time, and all event segments and the new event segment are used as target event segments;

[0156] A third determining unit, configured to determine a segment in the sliding window as a non-event segment when the fluctuation degree does not meet a preset fluctuation threshold;

[0157] The dividing unit is used to divide the target event segment into a long event segment and a short event segment according to the corresponding time length of the target event segment.

[0158] Further, the division module 201 includes an acquisition unit, a fourth determination unit, a fifth determination unit, a sixth determination unit and a seventh determination unit;

[0159] An acquiring unit is used to acquire a first preset formula, wherein the first preset formula satisfies:

[0160] ΔP=mean(W i POST )-mean(W i PRE ),

[0161] D(i)=[iw,i+w],

[0162] W i POST = {P i+1 ,P i+2 ,...,P i+w},

[0163] W i PRE = {P i-w ,P i-w+1 ,...,Pi-1},

[0164] ΔT=ji-2w,

[0165] ΔP is the fluctuation detection variable, ΔT is the time detection variable, D(i) is the sliding window, w is half of the preset length value, W i POST = {P i+1 ,P i+2 ,...,P i+w} is the data of the second half of the sliding window, W i PRE = {P i-w ,P i-w+1 ,...,P i-1} is the data of the first half of the sliding window, mean(·) is the function for finding the mean, i is the serial number of the first sliding window in the segment, and j is the serial number of the last sliding window in the segment;

[0166] A fourth determining unit, configured to determine that a segment in the sliding window is a non-event segment when the fluctuation detection variable is less than or equal to a preset fluctuation threshold;

[0167] A fifth determining unit, configured to determine the segment as an event segment when the fluctuation detection variable is greater than a preset fluctuation threshold;

[0168] a sixth determining unit, configured to determine that the segment is a short event segment when the time detection variable is less than or equal to a preset time value;

[0169] The seventh determination unit is used to determine that the segment is a long event segment when the time detection variable is greater than a preset time value.

[0170] Further, the acquisition module 202 includes a first compression unit, a second compression unit and a third compression unit;

[0171] a first compression unit, used for compressing the long event segment based on an improved fuzzy transformation algorithm when the target segment is a long event segment;

[0172] a second compression unit, configured to not compress the short event segment when the target segment is a short event segment;

[0173] The third compression unit is used to compress the non-event segment based on the SAX algorithm when the target segment is a non-event segment.

[0174] Further, the first compression unit includes a generation unit and a transformation unit;

[0175] A generating unit, used for generating a data matrix D∈N×M according to a long event segment;

[0176] A transformation unit, configured to perform a fuzzy transformation on a data matrix based on a second preset formula to obtain a target matrix, where the second preset formula satisfies:

[0177]

[0178] P = A T DB

[0179]

[0180]

[0181] F kl ∈n×m is the target matrix, P kl is the data matrix, Q kl is the fuzzy partition, I is the identity matrix of N×M, A = {A 1 ,..., A n} is the first fuzzy partition of the interval [1, N], B = {B 1 ,..., B m} is the second fuzzy partition of the interval [1, M], the fuzzy partition includes the first fuzzy partition and the second fuzzy partition, n < N, m < M, k ∈ (1, n), l ∈ (1, m), k is an integer, l is an integer, a is a positive integer less than the number of the first fuzzy partition, and b is a positive integer less than the number of the second fuzzy partition.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] As Figure 3 shown, an embodiment of the present invention further provides a device, including a processor 301, a communication interface 302, a memory 303, and a communication bus 304. Among them, the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0184] The memory 303 is used to store a computer program.

[0185] In an embodiment of the present invention, when the processor 301 is configured to execute the program stored on the memory 303, it implements the method provided in any one of the foregoing method embodiments, including:

[0186] Dividing the data into multiple sections of preset types according to the characteristics of the data;

[0187] Obtaining a target compression mode corresponding to the target segment to be compressed, wherein the target compression mode includes a preset compression mode corresponding to the type of the target segment;

[0188] Compress the target segment according to the target compression method.

[0189] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method provided in any one of the above method embodiments is implemented, including:

[0190] Divide the data into multiple preset types of segments according to the characteristics of the data;

[0191] Obtaining a target compression mode corresponding to the target segment to be compressed, wherein the target compression mode includes a preset compression mode corresponding to the type of the target segment;

[0192] Compress the target segment according to the target compression method.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0194] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0195] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A data compression method, characterized in that: include: According to the characteristics of the data, the data is divided into a plurality of preset types of segments; Acquire a target compression mode corresponding to the target segment to be compressed, wherein the target compression mode includes a preset compression mode corresponding to the type of the target segment; The target segment is compressed according to the target compression method.

2. The method according to claim 1, characterized in that: The types of segments include non-event segments, long event segments, and short event segments.

3. The method according to claim 1, characterized in that The step of dividing the data into a plurality of preset types of segments according to the characteristics of the data includes: Based on a sliding window with a preset number of data bits, determining a degree of fluctuation of data within the sliding window; When the fluctuation degree meets a preset fluctuation threshold, determining the segment within the sliding window as an event segment; When the time lengths corresponding to two adjacent event segments are less than a preset time, merging the two adjacent event segments into a new event segment; Repeat the step of merging the two adjacent event segments into a new event segment when the sum of the time lengths corresponding to the two adjacent event segments is less than the preset time, until the sum of the time lengths corresponding to all adjacent time periods and / or the new time period is greater than or equal to the preset time, and take all the event segments and the new event segment as the target event segment; When the fluctuation degree does not meet the preset fluctuation threshold, determining the segment within the sliding window as a non-event segment; According to the corresponding time length of the target event segment, the target event segment is divided into a long event segment and a short event segment.

4. The method according to claim 1, characterized in that: The step of dividing the data into a plurality of preset types of segments according to the characteristics of the data includes: Obtain a first preset formula, wherein the first preset formula satisfies: ΔP=mean(W i POST )-mean(W i PRE ), D(i)=[iw,i+w], W i POST ={P i+1 ,P i+2 ,...,P i+w }, W i PRE ={P i-w ,P i-w+1 ,...,P i-1 }, ΔT=ji-2w, ΔP is the fluctuation detection variable, ΔT is the time detection variable, D(i) is the sliding window, w is half of the preset length value, W i POST = {P i+1 ,P i+2 ,...,P i+w } is the data of the second half of the sliding window, W i PRE = {P i-w ,P i-w+1 ,...,P i-1 } is the data of the first half of the sliding window, mean(·) is the function for calculating the mean, i is the sequence number of the first sliding window in the segment, and j is the sequence number of the last sliding window in the segment; When the fluctuation detection variable is less than or equal to a preset fluctuation threshold, determining that the segment in the sliding window is a non-event segment; When the fluctuation detection variable is greater than the preset fluctuation threshold, determining the segment as an event segment; When the time detection variable is less than or equal to a preset time value, determining that the segment is a short event segment; When the time detection variable is greater than the preset time value, the segment is determined to be a long event segment.

5. The method according to any one of claims 2 to 4, characterized in that: The obtaining of a target compression mode corresponding to the target segment to be compressed includes: When the target segment is the long event segment, compressing the long event segment based on an improved fuzzy transformation algorithm; When the target segment is the short event segment, the short event segment is not compressed; When the target segment is the non-event segment, the non-event segment is compressed based on a SAX algorithm.

6. The method according to claim 5, characterized in that The long event segment is compressed based on an improved fuzzy transformation algorithm, including: According to the long event segment, a data matrix P is generated. kl ∈N×M; Based on a second preset formula, the data matrix is ​​fuzzy transformed to obtain a target matrix, wherein the second preset formula satisfies: P kl =A T DB, F kl ∈n×m is the target matrix, P kl is the data matrix, Q kl is the fuzzy partition, is the N×M identity matrix, A = {A1,..., A n} is the first fuzzy partition of the interval [1, N], B = {B1,..., B m} is the second fuzzy partition of the interval [1, M], the fuzzy partition includes the first fuzzy partition and the second fuzzy partition, n < N, m < M, k ∈ (1, n), l ∈ (1, m), k is an integer, l is an integer, a is a positive integer less than the number of the first fuzzy partition, and b is a positive integer less than the number of the second fuzzy partition.

7. A data compression device, characterized in that: include: A division module, used for dividing the data into a plurality of preset types of segments according to the characteristics of the data; An acquisition module, used for acquiring a target compression mode corresponding to the target segment to be compressed, wherein the target compression mode includes a preset compression mode corresponding to the type of the target segment; A compression module is used to compress the target segment according to the target compression method.

8. The device according to claim 7, characterized in that The division module includes a first determining unit, a second determining unit, a first merging unit, a second merging unit, a third determining unit and a division unit; The first determining unit is used to determine the degree of fluctuation of data in the sliding window based on a sliding window with a preset number of data bits; The second determining unit is configured to determine the segment in the sliding window as an event segment when the fluctuation degree meets a preset fluctuation threshold; The first merging unit is configured to merge the two adjacent event segments into a new event segment when the time lengths corresponding to the two adjacent event segments are less than a preset time; The second merging unit is used to repeat the step of merging the two adjacent event segments into a new event segment when the sum of the time lengths corresponding to the two adjacent event segments is less than the preset time, until the sum of the time lengths corresponding to all adjacent time periods and / or the new time period is greater than or equal to the preset time, and all the event segments and the new event segment are used as the target event segment; The third determining unit is configured to determine the segment in the sliding window as a non-event segment when the fluctuation degree does not meet the preset fluctuation threshold; The division unit is used to divide the target event segment into a long event segment and a short event segment according to the corresponding time length of the target event segment.

9. The device according to claim 7, characterized in that The division module includes an acquisition unit, a fourth determination unit, a fifth determination unit, a sixth determination unit and a seventh determination unit; The obtaining unit is used to obtain a first preset formula, wherein the first preset formula satisfies: ΔP=mean(W i POST )-mean(W i PRE ), D(i)=[iw,i+w], W i POST ={P i+1 ,P i+2 ,...,P i+w }, W i PRE ={P i-w ,P i-w+1 ,...,P i-1 }, ΔT=ji-2w, ΔP is the fluctuation detection variable, ΔT is the time detection variable, D(i) is the sliding window, w is half of the preset length value, W i POST = {P i+1 ,P i+2 ,...,P i+w } is the data of the second half of the sliding window, W i PRE = {P i-w ,P i-w+1 ,...,P i-1 } is the data of the first half of the sliding window, mean(·) is the function for calculating the mean, i is the sequence number of the first sliding window in the segment, and j is the sequence number of the last sliding window in the segment; The fourth determining unit is configured to determine that the segment in the sliding window is a non-event segment when the fluctuation detection variable is less than or equal to a preset fluctuation threshold; The fifth determining unit is configured to determine that the segment is an event segment when the fluctuation detection variable is greater than the preset fluctuation threshold; The sixth determination unit is configured to determine that the segment is a short event segment when the time detection variable is less than or equal to a preset time value; The seventh determination unit is configured to determine that the segment is a long event segment when the time detection variable is greater than the preset time value.

10. The device according to any one of claims 8 to 9, characterized in that: The acquisition module includes a first compression unit, a second compression unit and a third compression unit; The first compression unit is used for compressing the long event segment based on an improved fuzzy transformation algorithm when the target segment is the long event segment; The second compression unit is configured to, when the target segment is the short event segment, not compress the short event segment; The third compression unit is configured to compress the non-event segment based on a SAX algorithm when the target segment is the non-event segment.

11. The device according to claim 10, characterized in that The first compression unit includes a generation unit and a transformation unit; The generating unit is used to generate a data matrix D∈N×M according to the long event segment; The transformation unit is used to perform fuzzy transformation on the data matrix based on a second preset formula to obtain a target matrix, wherein the second preset formula satisfies: P=A T DB, F kl ∈ n×m is the target matrix, P kl is the data matrix, Qkl is the fuzzy partition, is the N×M identity matrix, A = {A1,..., A n} is the first fuzzy partition of the interval [1, N], B = {B1,..., B m} is the second fuzzy partition of the interval [1, M], the fuzzy partition includes the first fuzzy partition and the second fuzzy partition, n < N, m < M, k ∈ (1, n), l ∈ (1, m), k is an integer, l is an integer, a is a positive integer less than the number of the first fuzzy partition, and b is a positive integer less than the number of the second fuzzy partition.

12. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data compression method according to any one of claims 1 to 6.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data compression method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Multi-feature adaptive data compression method and system for power distribution network terminal

    CN122068907A