A data real-time compression storage method, medium and system

Abstract

The application discloses a data real-time compression method, medium and system, and the method comprises the following steps: collecting one data at each collection time; starting from the first data collected this time, adopting a rotating door trend algorithm to perform data compression storage this time; the step of adopting the rotating door trend algorithm to perform data compression storage this time comprises the following steps: during the process of compression storage this time, starting from the second data collected, calculating a first standard value and a second standard value corresponding to the current data; calculating the sum of the absolute value of the first standard value and the absolute value of the second standard value corresponding to the current data to obtain a judgment value corresponding to the current data; comparing the judgment value corresponding to the current data with the size relationship between the data abnormal storage value and the data normal storage value; and performing data compression storage this time according to the size relationship. The application can realize lossless compression storage of data, and solves the problems of large data volume and slow data access from the root.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, medium, and system for real-time data compression and storage. Background Technology

[0002] Substations or digital converter stations have online monitoring systems that connect to a large number of monitored devices, including oil-filled, gas-filled, surge arresters, bushings, and other critical equipment. Each monitored device contains a certain number of monitoring points, resulting in a large volume of real-time collected data. Currently, the collected online monitoring data is stored directly into the database without processing. As the online monitoring system operates for an extended period, the system's data storage capacity will gradually increase, leading to slow data access and retrieval, difficulties in data analysis and feature extraction, and hindering effective management of the monitored equipment's status. Summary of the Invention

[0003] This invention provides a real-time data compression and storage method, medium, and system to solve the problem that data collected in the prior art is directly stored in the database without processing, which leads to a gradual increase in data storage volume and slow data access.

[0004] Firstly, a real-time data compression method is provided, including:

[0005] Collect one data point at each acquisition time;

[0006] Starting with the first data point collected, the revolving door trend algorithm is used to compress and store the data.

[0007] The step of using the revolving door trend algorithm for data compression and storage includes:

[0008] During this compression and storage process, starting from the second data acquisition, the first and second standard values ​​corresponding to the current data are calculated;

[0009] Calculate the sum of the absolute values ​​of the first and second standard values ​​corresponding to the current data to obtain the judgment value corresponding to the current data;

[0010] Compare the current data's corresponding judgment value with the data's abnormal storage value and normal storage value;

[0011] The data is compressed and stored according to the aforementioned size relationship.

[0012] In a second aspect, a computer-readable storage medium is provided, wherein computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, they implement the real-time data compression and storage method as described in the first aspect embodiment above.

[0013] Thirdly, a real-time data compression and storage system is provided, comprising: a computer-readable storage medium as described in the second aspect embodiment above.

[0014] Thus, this embodiment of the invention can achieve lossless compressed storage of collected data, fundamentally solving problems such as large data volume and slow data access, and providing convenience for subsequent system data analysis and data feature extraction. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a real-time data compression and storage method according to an embodiment of the present invention;

[0017] Figure 2 This is a flowchart of the revolving door trend algorithm according to an embodiment of the present invention;

[0018] Figure 3 This is a comparison diagram of data compression before and after a specific embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention discloses a real-time data compression and storage method. This method can be applied to the compression and storage of online monitoring data in substations. Figure 1 As shown, the method includes the following steps:

[0021] Step S101: Collect one data point at each acquisition time.

[0022] Generally, the time interval between any two adjacent acquisition times is equal, for example, an interval of 5 seconds.

[0023] This data can be collected in real time sequentially, or it can be historical data acquired sequentially.

[0024] When the data is collected in real time, the collected data can be transmitted through industrial networks and transmission standards so that some new collected data that has changed can be processed by the method of the embodiments of the present invention.

[0025] When the data is historical data, it can be obtained through the computer file system and corresponding database applications so that the original data can be processed by the method of the present invention, so that the validity of the data before and after compression remains consistent.

[0026] Step S102: Starting from the first data point collected this time, the rotating door trend algorithm is used to compress and store the data.

[0027] Specifically, such as Figure 2 As shown, this step includes the following process:

[0028] Step S201: During this compression and storage process, starting from the second data acquisition, calculate the first standard value and the second standard value corresponding to the current data.

[0029] In other words, the compression and storage process starts from the acquisition of the second piece of data.

[0030] The process of calculating the first and second standard values ​​for the second data collected during this compression and storage process is as follows:

[0031] (1) If the current data is the second data collected in this compression and storage process, then calculate the first initial standard value and the second initial standard value of the second data respectively.

[0032] (2) If the first initial standard value of the second data is less than 0, then the first standard value of the second data is determined to be the sum of the first initial standard value of the second data and the preset threshold; otherwise, the first standard value of the second data is determined to be the first initial standard value of the second data.

[0033] (3) If the second initial standard value of the second data is less than 0, then the second standard value of the second data is determined to be the sum of the second initial standard value of the second data and the preset threshold; otherwise, the second standard value of the second data is determined to be the second initial standard value of the second data.

[0034] The preset threshold can be selected based on experience. In this embodiment of the invention, the preset threshold is determined to be 1.57, then K... t2a K represents the first standard value of the second data collected during this compression and storage process. t2b K represents the second standard value of the second data collected during this compression and storage process. t2a,0 K represents the first initial standard value of the second data collected during this compression and storage process. t2b,0Let the second initial standard value represent the second data collected in this compression and storage process, then we get:

[0035] If K t2a,0 <0, then K t2a =K t2a,0 +1.57; if K t2a,0 If K ≥ 0, then t2a =K t2a,0 .

[0036] If K t2b,0 <0, then K t2b =K t2b,0 +1.57; if K t2b,0 If K ≥ 0, then t2b =K t2b,0 .

[0037] Specifically, for the data collected after the second data point during this compression and storage process, the process for calculating the first and second standard values ​​is as follows:

[0038] (1) Starting from the third data collected in this compression and storage process, calculate the first initial standard value and the second initial standard value of the current data.

[0039] (2) If the first initial standard value of the current data is greater than the first standard value of the previous data, then the first standard value of the current data is determined to be the first initial standard value of the current data; otherwise, the first standard value of the current data is determined to be the first standard value of the previous data.

[0040] With K tna,0 K represents the first initial standard value of the nth data point collected during this compression and storage process. t(n-1)a K represents the first standard value of the (n-1)th data point collected during this compression and storage process. tna This represents the first standard value of the nth data point collected during this compression and storage process. If K tna,0 >K t(n-1)a Then K tna =K tna,0 If K tna,0 ≤K t(n-1)a Then K tna =K t(n-1)a It should be understood that, at this point, n≥3.

[0041] (3) If the second initial standard value of the current data is greater than the second standard value of the previous data, then the second standard value of the current data is determined to be the second initial standard value of the current data; otherwise, the second standard value of the current data is determined to be the second standard value of the previous data.

[0042] With K tnb,0K represents the second initial standard value of the nth data point collected during this compression and storage process. t(n-1)b K represents the second standard value of the (n-1)th data point collected during this compression and storage process. tnb This represents the second standard value of the nth data point collected during this compression and storage process. If K tnb,0 >K t(n-1)b Then K tnb =K tnb,0 If K tnb,0 ≤K t(n-1)b Then K tnb =K t(n-1)b It should be understood that, at this point, n≥3.

[0043] Specifically, regardless of which data point is being collected, the formula for the first initial standard value is:

[0044]

[0045] The calculation results are expressed in radians.

[0046] Regardless of which data point is being collected, the formula for the second initial standard value is:

[0047]

[0048] The calculation results are expressed in radians.

[0049] Specifically,

[0050]

[0051] c t1 =2U.

[0052] Among them, t n t1 represents the acquisition time corresponding to the nth data point collected in this compression and storage process, in seconds; t2 represents the acquisition time corresponding to the first data point collected in this compression and storage process, in seconds; x tn x represents the nth data point collected during this compression and storage process. t1 This represents the first data collected during this compression and storage process. It should be understood that at this point, n≥2.

[0053] U represents the real-time data acquisition deviation value. Generally, U is a real number greater than 0, and the larger the U, the lower the data density. Therefore, the basic principle for setting the U value is that the greater the data change, the larger the U value. The U value can be appropriately adjusted according to the actual data characteristics of the monitored equipment. In this embodiment of the invention, U = 1 is selected.

[0054] Step S202: Calculate the sum of the absolute value of the first standard value and the absolute value of the second standard value corresponding to the current data to obtain the judgment value corresponding to the current data.

[0055] That is, f(x tn ) = |K tna | + |K tnb |.

[0056] Among them, f(x tn ) represents the judgment value corresponding to the nth data collected in this compression storage process.

[0057] Step S203: Compare the size relationship between the judgment value corresponding to the current data and the data abnormal storage value and the data normal storage value.

[0058] Among them, the data abnormal storage value and the data normal storage value can both be preset according to experience. Let K1 represent the data abnormal storage value and K2 represent the data normal storage value. In the embodiment of the present invention, K1 = 3.5 and K2 = 3.14.

[0059] Step S204: Perform compression storage of this data according to the size relationship.

[0060] Specifically, this step includes the following three situations:

[0061] (1) If the judgment value corresponding to the current data is not less than the data abnormal storage value, then save the previous data, the current data, and the preset number of data after the current data.

[0062] If it satisfies |K tna | + |K tnb | ≥ K1, it indicates that the data is abnormal, and corresponding storage is performed to save the characteristic data.

[0063] The preset number can be selected according to experience. For example, the preset number is 20, and the current data is the second data collected in this compression storage process, then save the first data to the twenty-first data collected in this compression storage.

[0064] (2) If the judgment value corresponding to the current data is not less than the data normal storage value and less than the data abnormal storage value, then save the previous data of the current data.

[0065] If it satisfies K2 ≤ |K tna | + |K tnb | < K1, it indicates that the corresponding data is normal and is indispensable for data analysis, feature extraction, etc., then store the corresponding data.

[0066] For example, if the current data is the second data collected in this compression storage process, then save the first data collected in this compression storage.

[0067] (3) If the judgment value corresponding to the current data is less than the normal data storage value, the current data is discarded.

[0068] If it satisfies |K tna | + |K tnb | < K2, it indicates that deleting the corresponding data does not affect data analysis, feature extraction, etc. Therefore, the corresponding data can be deleted and the data can be compressed to reduce the amount of stored data.

[0069] For example, if the current data is the second data collected in this compression storage process, the second data is discarded, that is, the second data will not be stored.

[0070] Specifically, after the steps of saving the previous data, the current data, and the preset number of data after the current data, the method of the embodiment of the present invention further includes:

[0071] Taking the next data after the preset number of data after the current data as the first data for the next collection, and using the rotating door trend algorithm to perform the next data compression storage.

[0072] For example, if the preset number is 20 and the current data is the second data collected in this compression storage process, the first data to the twenty - first data collected in this compression storage are saved, and the twenty - second data is used as the first data for the next data compression storage.

[0073] Specifically, after the step of saving the previous data of the current data, the method of the embodiment of the present invention further includes:

[0074] Taking the current data as the first data for the next collection, and using the rotating door trend algorithm to perform the next data compression storage.

[0075] For example, if the current data is the second data collected in this compression storage process, the first data collected in this compression storage is saved, and the second data is used as the first data for the next data compression storage.

[0076] It should be understood that no matter which data is used as the starting point for the next compression storage, the compression storage process is carried out according to the foregoing process.

[0077] Specifically, after the step of discarding the current data, the method of the embodiment of the present invention further includes:

[0078] Continuing the current data compression storage process.

[0079] For example, if the current data is the second data collected in this compression storage process, the second data is discarded, and the judgment value corresponding to the third data is continuously calculated, and the corresponding compression storage operation is performed according to the judgment value.

[0080] Tests have shown that the above-mentioned compression storage method can increase storage space by 23.6%.

[0081] After storing the data, when data needs to be read, if there is corresponding stored data for the selected collection time, it can be read directly. If there is no corresponding stored data for the selected collection time, preferably, the method of this embodiment of the invention achieves data reading in the following manner:

[0082] (1) During the process of reading the stored data, if the selected collection time does not store data, the search is performed backward from the selected collection time to obtain the earliest collection time that stores data and the closest collection time to the selected collection time. Then, the search is performed backward from the selected collection time to obtain the latest collection time that stores data and the closest collection time to the selected collection time.

[0083] (2) Establish the first data reading equation.

[0084] Specifically, the first data reading equation is: x = kt + p.

[0085] Where x represents the data, t represents time, k represents the slope, and p represents the intercept.

[0086] (3) The slope of the first data reading equation corresponding to the selected acquisition time is calculated by using the most recent prior acquisition time and the corresponding data, as well as the most recent subsequent acquisition time and the corresponding data.

[0087] Specifically, the slope calculation formula is:

[0088] Where, x f This represents the data corresponding to the most recent prior collection time, x. b t represents the data corresponding to the most recent collection time. f Indicates the most recent prior acquisition time, t b The most recent data collection time.

[0089] (4) Establish the second data reading equation.

[0090] Specifically, the second data reading equation is:

[0091] x = kt.

[0092] (5) Substitute the selected acquisition time and the slope corresponding to the selected acquisition time into the second data reading equation to calculate the data corresponding to the selected acquisition time.

[0093] In this way, the discarded data can be read using the data reading method described above.

[0094] An embodiment of the present invention also discloses a computer-readable storage medium, on which computer program instructions are stored; when the computer program instructions are executed by a processor, the data real-time compression storage method described in the above embodiment is implemented.

[0095] An embodiment of the present invention also discloses a data real-time compression storage system, including: the computer-readable storage medium described in the above embodiment.

[0096] The technical solution of the present invention will be further described below with a specific embodiment.

[0097] Taking the historical data of the ground current of a certain converter transformer core as an example, the data is retrieved from the server through an online monitoring platform, as shown in Table 1.

[0098] Table 1 Historical data of the ground current of a certain converter transformer core

[0099]

[0100]

[0101]

[0102] K1 = 3.5, K2 = 3.14, U = 1.

[0103] When n = 2, t2 = 5, t1 = 0, x t1 = 42.8823, x t2 = 43.2885, and a t2 = 5.193977131, b t2 = 5.035136387, c t1 = 2 are calculated, and K t2a,0 = 1.45, K t2b,0 t2b,0 = 1.30 are obtained. Since K t2a,0 > 0, K t2b,0 > 0, then K t2a = 1.45, K t2b = 1.30, |K t2a | + |K t2b | = 2.75 <K2, so x t2 is discarded.

[0104] And so on, when n = 3, t3 = 10, x t3 = 41.6416, and K t3a,0 = 1.35, K t3b,0 = 1.60 are calculated. Since K t3a,0 < K t2a , K t3b,0 > K t2b , then Kt3a = 1.45, K t3b = 1.60, |K t3a | + |K t3b | = 3.05 < K2, then discard x t3 .

[0105] And so on. When n = 4, t₄ = 15, x t4 = 40.1343, calculate to get K t4a,0 = 1.33, K t4b,0 = 1.69. Since K t4a,0 < K t3a , K t4b,0 > K t3b , then K t4a = 1.45, K t4b = 1.69, |K t4a | + |K t4b | = 3.15, K2 < 3.15 < K1, then save x t3 .

[0106] The process of this data compression and storage is completed. Then use x t4 as the first data for the next data compression and storage, starting from x t4 to start the process of the next compression and storage.

[0107] Taking some collected data as an example, its compression processing results are as Figure 3 shown. It can be seen from Figure 3 that the validity of the data before and after compression remains consistent.

[0108] In summary, the embodiments of the present invention can unify the data format, compress and store normal changing data, store abnormal changing data normally, guarantee the data characteristics and data storage space, solve problems such as large data volume and slow data access and storage, improve work efficiency and response speed, and are convenient for operation and maintenance personnel to obtain characteristic data and abnormal data curves, which has very important significance for online monitoring of real-time data compression, access, analysis and feature extraction.

[0109] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method of data real-time compression, characterized in that, The method comprises the following steps: collecting one data at each collection time; starting from the first data collected this time, performing this-time data compression storage by using a rotating door trend algorithm; wherein the step of performing this-time data compression storage by using a rotating door trend algorithm comprises the following steps: during the this-time compression storage process, starting from the second data collected, calculating a first standard value and a second standard value corresponding to the current data; calculating the sum of the absolute value of the first standard value and the absolute value of the second standard value corresponding to the current data to obtain a judgment value corresponding to the current data; comparing the judgment value corresponding to the current data with the size relationship between the data abnormal storage value and the data normal storage value; performing this-time data compression storage according to the size relationship; the step of calculating the first standard value and the second standard value corresponding to the current data comprises the following steps: if the current data is the second data collected in the this-time compression storage process, then calculate a first initial standard value of the second data and a second initial standard value of the second data respectively; if the first initial standard value of the second data is less than 0, then determine the first standard value of the second data as the sum of the first initial standard value of the second data and a preset threshold value, otherwise, determine the first standard value of the second data as the first initial standard value of the second data; if the second initial standard value of the second data is less than 0, then determine the second standard value of the second data as the sum of the second initial standard value of the second data and a preset threshold value, otherwise, determine the second standard value of the second data as the second initial standard value of the second data; starting from the third data collected in the this-time compression storage process, calculate the first initial standard value and the second initial standard value of the current data; if the first initial standard value of the current data is greater than the first standard value of the previous data, then determine the first standard value of the current data as the first initial standard value of the current data, otherwise, determine the first standard value of the current data as the first standard value of the previous data; if the second initial standard value of the current data is greater than the second standard value of the previous data, then determine the second standard value of the current data as the second initial standard value of the current data, otherwise, determine the second standard value of the current data as the second standard value of the previous data; the step of performing this-time data compression storage according to the size relationship comprises the following steps: if the judgment value corresponding to the current data is not less than the data abnormal storage value, then save the previous data of the current data, the current data, and a preset number of data after the current data; if the judgment value corresponding to the current data is not less than the data normal storage value and less than the data abnormal storage value, then save the previous data of the current data; if the judgment value corresponding to the current data is less than the data normal storage value, then discard the current data.

2. The data real-time compression method according to claim 1, wherein after the step of saving the previous data of the current data, the current data, and a preset number of data after the current data, the method further comprises the following steps: An algorithm of the first initial criterion value is: ; The formula of the second initial criterion value is: ; wherein, represents the first initial standard value of the nth data collected in the current compression storage process, n represents the second initial standard value of the nth data collected in the current compression storage process;​ ； ； ； wherein, represents the acquisition time corresponding to the first data collected in the current compression storage process, n represents the acquisition time corresponding to the first data collected in the current compression storage process, represents the acquisition time corresponding to the first data collected in the current compression storage process, represents the first data collected in the current compression storage process, n represents the first data collected in the current compression storage process, represents the first data collected in the current compression storage process, U represents the real-time data acquisition deviation value, n ≥ 2.

3. The data real-time compression storage method according to claim 1, wherein, taking the next data of the preset number of data after the current data as the first data collected next time, and performing next-time data compression storage by using a rotating door trend algorithm. ​ 4. The data real-time compression storage method of claim 1, wherein, After the step of saving the previous data of the current data, the method further comprises: Using the current data as the first data of the next acquisition, using the rotating door trend algorithm to perform the next data compression storage.

5. The data real-time compression storage method of claim 1, wherein, After the step of discarding the current data, the method further comprises: Continuing the process of the current data compression storage.

6. The data real-time compression storage method of claim 1, wherein, Further comprising: In the process of reading the stored data, if the selected acquisition time has no stored data, then searching forward from the selected acquisition time to obtain the nearest previous acquisition time with stored data, and searching backward from the selected acquisition time to obtain the nearest subsequent acquisition time with stored data; establishing a first data reading equation, wherein the first data reading equation is: , x represents data, t represents time, k represents a slope, p represents an intercept; calculating a slope of the first data reading equation corresponding to a selected acquisition time from the most recent preceding acquisition time and corresponding data, and the most recent succeeding acquisition time and corresponding data, wherein the slope calculation is: , denotes data corresponding to the most recent preceding acquisition time, denotes data corresponding to the most recent succeeding acquisition time, denotes the most recent preceding acquisition time, the most recent succeeding acquisition time; establishing a second data reading equation, wherein the second data reading equation is: ; Substituting the selected acquisition time and the slope corresponding to the selected acquisition time into the second data reading equation to calculate the data corresponding to the selected acquisition time.

7. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer program instructions; the computer program instructions are executed by the processor to realize the data real-time compression storage method in any one of claims 1-6.

8. A data real-time compression storage system, characterized by, Comprise: The computer readable storage medium of claim 7.

Images