A single-point lossy compression method and device for industrial Internet of Things real-time database
By adopting lossy compression methods of dynamic prediction and correction in real-time database of industrial IoT, the slope limit rate and correction value are used to calculate the upper and lower slope limits of new data, efficient data compression is achieved, and the compression rate in the existing technology is solved and the problem of improving accuracy space, suitable for scenarios with high redundancy and high real-time requirements.
Patent Information
- Application Number
- CN202111264660.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-10-28
AI Technical Summary
Lossy compression algorithms in existing real-time databases are difficult to meet the high compression ratio and high accuracy requirements of industrial IoT data. Especially in scenarios where data redundancy is high and real-time requirements are high, there is more room for improvement in compression rate and accuracy. .
Through dynamic prediction, correction and inspection methods, the data in the real-time database of the Industrial Internet of Things is lostly compressed, and the slope limit rate and correction value are used to calculate the upper and lower slope limits of the new data, dynamically adjust the compression range, and achieve efficient data Loss compression.
Improve the compression rate and accuracy of lossy compression, meet the storage needs of real-time industrial IoT databases, ensure that the system maintains restoration accuracy while having high compression ratios, and reduces human intervention factors, which are suitable for generating large amounts of data in a short time. Scene.
Smart Images

Figure CN114142865B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method in the field of real-time database, and in particular to a method and device for lossy compression of industrial Internet of Things real-time database data through dynamic prediction, correction and verification. Background Art
[0002] With the rapid development of the Industrial Internet of Things, the amount of data that needs to be processed and stored has increased dramatically. These data are of great significance for the analysis of production conditions and the monitoring of equipment performance. These data are recorded in the database, thus generating a real-time database. In view of the problems that real-time systems generate a large amount of data in a short period of time, there is a lot of data redundancy, and the real-time requirements are very high, it is very necessary to introduce data compression technology into the real-time database system, and finding a compression algorithm with high compression ratio and high compression and decompression speed is the key.
[0003] At present, real-time database compression technology can be mainly divided into lossy compression and lossless compression. Lossy compression is an irreversible compression. Its principle is to selectively retain some data and discard some data through certain data screening rules, and ensure that the restoration accuracy of the decompression algorithm is within the allowable range of system accuracy error. Lossless compression is also called redundancy compression. It is a reversible compression. Its essence is to reduce the redundant components in the source data, but it cannot discard the valid information in the source data, so as to ensure that the source data before compression and the data after decompression and restoration are completely consistent.
[0004] Lossless compression is subject to theoretical limitations of data statistical redundancy, and its compression ratio is generally not too high, which cannot solve all data storage and transmission problems. Therefore, it is mainly used in situations that require accurate data recovery and distortion-free information transmission. On the contrary, lossy compression algorithms have a higher compression ratio and can be applied to the storage needs of real-time database systems, but at the same time, they will also lead to a decrease in system accuracy.
[0005] At present, common lossy compression algorithms include dead zone limit compression algorithm and trend compression algorithm. The dead zone limit compression algorithm compares the new value with the last recorded value. When the absolute value of the difference between the two is less than the error threshold, the data is discarded. Otherwise, the data is retained. The dead zone compression algorithm is simple, but the compression ratio is low, which makes it difficult to meet the requirements of real-time database systems. Trend compression is represented by PI's revolving door algorithm. The revolving door algorithm improves the compression rate of real-time databases, but does not describe, record, or predict the changes in trends. Therefore, there is still room for improvement in compression rate and accuracy.
[0006] After searching, it was found that the Chinese patent with publication number CN108197181A disclosed a compression storage method, electronic device and storage medium for time series data on June 22, 2018, including an allocation step, a table building step, a merging step and a storage step. The patent compresses the time series data belonging to the same row in the measurement point data table. The revolving door algorithm is a linear fitting algorithm that can reduce the amount of time series data stored and save storage space. However, there is still a problem that the changes in the compression trend cannot be described and recorded. Therefore, in view of the data characteristics of the industrial Internet of Things real-time database and the requirements for compression algorithms, those skilled in the art still need to conduct in-depth research on lossy compression methods in industrial Internet of Things real-time databases. Summary of the invention
[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and device for lossy compression in an industrial Internet of Things real-time database. By dynamically predicting, correcting and verifying the method for lossy compression of real-time data, the compression rate and accuracy of lossy compression can be improved at the same time.
[0008] The present invention is achieved through the following technical solutions:
[0009] A single-point lossy compression method for an industrial Internet of Things real-time database comprises the following steps:
[0010] Collect data based on a preset time period and perform data preprocessing on the collected data;
[0011] Calculate the slopes between the first k preprocessed data, obtain the upper and lower limits of the current slope based on the preset error tolerance range and sort them, and obtain the current slope limit rate, determine the data corresponding to the slope limit rate as the slope limit rate point, write the data before the slope limit rate point into the database, and record the slope limit rate point as the "last written data", and the kth data as the "last read data";
[0012] Read in new data, calculate a new slope for the new data and the "last written data" based on the preset initial correction value; adjust the current slope limit rate at the same time to obtain the slope limit rate range of the new data; combine the new slope and the slope limit rate range to obtain the upper and lower limits of the slope of the new data;
[0013] Compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope. If the upper and lower limits of the slope of the new data are both greater than the upper limit of the current slope or the upper and lower limits of the slope of the new data are both less than the lower limit of the current slope, save the new data and use the new data as the updated slope limit rate point. Record the data of the updated slope limit rate point as the "last read data";
[0014] Calculate the slope limit rate corresponding to the updated slope limit rate point as the updated current slope limit rate; calculate the slope correction value at the same time, and calculate the slope upper and lower limits based on the slope correction value as the updated current slope upper and lower limits;
[0015] Read in new data again and repeat the calculation and judgment steps until all data compression is completed.
[0016] In the above technical scheme, initial data is collected and preprocessed, the initial slope is calculated to obtain the current upper and lower limits of the slope, the initial slope limit rate and the slope limit rate point are defined, and the data is written to the database; the slope of the new data is calculated and the current slope limit rate is adjusted, and the upper and lower limits of the slope of the new data are obtained after the slope limit rate interval is obtained; whether to save the new data is determined by comparing the upper and lower limits of the slopes of the current and new data. If the new data is to be saved, the slope limit rate point is updated and the slope limit rate is calculated; the current upper and lower limits of the slope are adjusted, and the steps of determining whether to save the new data are repeated, thereby realizing lossy compression of real-time data through dynamic prediction, correction and verification, while improving the compression rate and accuracy of the lossy compression.
[0017] Preferably, the data preprocessing process includes: reading in k real-time data P1, P2, ..., P k , and perform normalization processing, and define the normalization processing formula of the i-th (i∈k) real-time collected data as follows:
[0018]
[0019] Where P i is the i-th real-time data collection, p i is the normalized data, P min is the minimum value of the data, P max is the maximum value of the data. After the data is normalized, it is arranged in time series to obtain the processed data: p1(t1,v1), p2(t2,v2), ..., p k (t k ,v k ), where t1, t2, … t k Indicates the time value of the collected data, v1, v2, …v k The data value representing the collected data.
[0020] Specifically, normalizing the input data can improve the convergence speed of the data model and improve the accuracy of the data model. Normalizing the input data can make the features between different dimensions comparable in numerical value, which can greatly improve the accuracy of the classifier.
[0021] Preferably, the process of calculating the slope of the first k initial data includes:
[0022] S11, traverse the data, based on the preset error tolerance range, calculate the slope limit of the data pairwise, and obtain The upper slope limit, The lower slope limit and The slope of the midline.
[0023] Specifically, taking p1 and p2 as examples, the upper limit of the slope is: r up1 =(v2+T-v1) / (t2-t1), the lower limit of the slope is: r down1 =(v2-T-v1) / (t2-t1), the slope midline is: r1=(v2-v1) / (t2-t1), we get The slope upper limit, slope lower limit and slope center line are calculated, where T is the error tolerance range ±T, and the slope correction value d is initialized. r is 0.
[0024] S12, the result obtained in S11 Sort the upper and lower limits of the slope from small to large, and take the average of the upper limits of the slope. is the upper limit of the slope r up , the average value of the lower limit of the slope is the lower limit of the slope r down .
[0025] S13, the result obtained in S11 The slope midlines r are sorted from small to large, and the initial slope limit rate α is selected from the sorted slope midlines according to the preset ratio. p , take the data corresponding to the initial slope limit rate as the initial slope limit rate point p α .
[0026] S14, recording the initial slope limit rate point as the "last written data", and recording the kth data as the "last read data".
[0027] Specifically, p α Recorded as "last written data" as p last (t last ,v last ), p k Recorded as "last read data" as p read (t read ,v read ).
[0028] Preferably, the process of calculating a new slope using new data and "last written data" includes:
[0029] S21. Read in new data and perform normalization processing.
[0030] Specifically, for the new data P k+1After normalization, we get p k+1 (t k+1 ,v k+1 ).
[0031] S22. Calculate the slope limit of the new data according to the "last written data", calculate a new upper slope limit, and calculate a new lower slope limit.
[0032] Specifically, calculate the upper limit of the slope of the new data as r' up1 =(v k+1 +Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last ), calculate the lower limit of the slope of the new data as r' down1 =(v k+1 -Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last ).
[0033] S23, set the adjustment interval to λ, and calculate the adjusted slope limit rate: α p ±λσ, and sort them from small to large to form the slope limit rate interval of the point, taking the maximum and minimum values of the interval, where λ is the adjustment interval.
[0034] Specifically, 1≤λ≤m, m is p k+1 With p α The number of interval data, the initial σ is 0.05.
[0035] S24. The smaller of the new upper slope limit and the interval maximum value is taken as the upper slope limit of the new data, and the smaller of the new lower slope limit and the interval minimum value is taken as the lower slope limit of the new data.
[0036] Specifically, take the upper limit r' of the slope of the new data up =min{r' up1 ,r' up2}, take the lower limit of the slope of the new data r' down =max{r' down1 ,r' down2}.
[0037] Preferably, the upper and lower limits of the slope of the new data are compared with the upper and lower limits of the current slope. If either of the upper and lower limits of the slope of the new data is not greater than the upper limit of the current slope or either of the upper and lower limits of the slope of the new data is not less than the lower limit of the current slope, it is determined not to save the new data.
[0038] Specifically, if r' up >r up And r' down >r up or r' down <r down And r' up <r down , then you need to save the new data, otherwise do not save the data.
[0039] Specifically, S41, if new data needs to be saved, calculate the slope correction value:
[0040] d r =((v k+1 -v read ) / (t k+1 -t read )-r) / (t read -t last ).
[0041] S42, draw a vertical line through the "last read data", if r' up >r up , and the vertical line is connected to r up The intersection is written into the database and recorded as the "last written data". If r' down <r down , and the vertical line is connected to r down The intersection point is written into the database and recorded as the "last written data", and the upper and lower limits are recalculated.
[0042] r up =(v k+1 +Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last );
[0043] r down =(v k+1 -Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last ).
[0044] Preferably, if new data is not saved, the current upper and lower limits of the slope are readjusted, and the specific rules for adjusting the current upper and lower limits of the slope are: if the upper limit of the slope of the new data is less than the upper limit of the current slope, then the upper limit of the slope of the new data is adjusted as the upper limit of the current slope after the update; if the lower limit of the slope of the new data is greater than the lower limit of the current slope, then the lower limit of the slope of the new data is adjusted as the lower limit of the current slope after the update.
[0045] Specifically, if there is no need to save new data, if r' up <r up , adjust r up for r' up , if r' down >r down , adjust r down for r' down .
[0046] Preferably, if new data is saved, the slope midline of the slope limit rate point is calculated as the updated current slope limit rate.
[0047] Preferably, no matter whether the new data is saved or not, the new data is recorded as the "last read data".
[0048] Furthermore, the decompression process includes the following steps: Assume that the query time is t, then read the three most recent data before t, p1(t1,v1), p2(t2,v2), p3(t3,v3), and calculate p t (t,v t ) is as follows: k1 = (v2-v1) / (t2-t1), k2 = (v3-v2) / (t3-t2), v t =v3+(t3-t2)(k2+(k2-k1)(t3-t2) / (t2-t1)).
[0049] An industrial Internet of Things real-time database single-point lossy compression device, comprising:
[0050] A data acquisition module is used to collect data based on a preset time period and perform data preprocessing on the collected data;
[0051] The data initialization module is used to calculate the slopes between the first k pre-processed data, obtain the upper and lower limits of the current slope based on the preset error tolerance range and sort them, and obtain the current slope limit rate, determine the data corresponding to the slope limit rate as the slope limit rate point, write the data before the slope limit rate point into the database, and record the slope limit rate point as the "last written data", and the kth data as the "last read data";
[0052] The data calculation module is used to read in new data, calculate a new slope for the new data and the "last written data" based on a preset initial correction value; adjust the current slope limit rate at the same time to obtain a slope limit rate interval for the new data; and obtain the upper and lower limits of the slope of the new data by combining the new slope and the slope limit rate interval;
[0053] The data comparison and saving module is used to compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope. If the upper and lower limits of the slope of the new data are both greater than the upper limit of the current slope or the upper and lower limits of the slope of the new data are both less than the lower limit of the current slope, the new data is saved and the new data is used as the updated slope limit rate point, and the data of the updated slope limit rate point is recorded as the "last read data";
[0054] The data update module is used to calculate the slope limit rate corresponding to the updated slope limit rate point as the updated current slope limit rate; at the same time, the slope correction value is calculated, and the slope upper and lower limits are calculated based on the slope correction value as the updated current slope upper and lower limits.
[0055] Preferably, the data updating module is further used to readjust the current upper and lower limits of the slope when no new data is saved.
[0056] Compared with the prior art, the present invention has the following beneficial effects:
[0057] (1) The present invention collects initial data, performs preprocessing, calculates the initial slope to obtain the current slope upper and lower limits, defines the initial slope limit rate and the slope limit rate point, and writes the data into a database; calculates the slope of the new data and adjusts the current slope limit rate, obtains the slope upper and lower limits of the new data after obtaining the slope limit rate interval; determines whether to save the new data based on the comparison of the slope upper and lower limits of the current and new data, and if the new data is to be saved, updates the slope limit rate point and calculates the slope limit rate; adjusts the current slope upper and lower limits, and repeats the step of determining whether to save the new data, thereby achieving lossy compression of real-time data through dynamic prediction, correction and verification, while improving the compression rate and accuracy of the lossy compression;
[0058] (2) The present invention aims at solving the problems that the real-time system of the Industrial Internet of Things generates a large amount of data in a short period of time, has a lot of data redundancy, and has high requirements for real-time performance, and the compression ratio and restoration accuracy of the current lossless compression algorithm still have room for improvement. Through dynamic correction and prediction, the real-time data is quickly and efficiently lossy compressed, which can improve the compression ratio while meeting the system's requirements for restoration accuracy. It can be applied to the storage needs of real-time databases in the Industrial Internet of Things;
[0059] (3) The present invention adopts a method of regularly updating and confirming the σ value, slope limit value and interval, so that the data compression accuracy is dynamically adjusted and automatically adapted as the slope limit rate interval and new data change, effectively reducing human intervention factors, providing conditions for the subsequent expansion of AI (artificial intelligence) compression, and ensuring that the system is in a reliable and efficient operation state for a long time. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 A flowchart of a lossy compression method in a real-time database according to an embodiment of the present invention;
[0061] Figure 2 A diagram showing the concept definition of a slope limit rate according to an embodiment of the present invention;
[0062] Figure 3 A flow chart of a method for periodically diagnosing and updating the working status of a monitoring point according to an embodiment of the present invention;
[0063] Figure 4 is a flow chart of a decompression process according to an embodiment of the present invention;
[0064] Figure 5 4 is a system framework diagram according to an embodiment of the present invention. DETAILED DESCRIPTION
[0065] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0066] Example 1
[0067] This embodiment proposes a lossy compression method in an industrial Internet of Things real-time database. First, read in the agreed number of continuous real-time data, and calculate the initial slope upper and lower limits and the slope limit rate according to the corresponding real-time collected data and the allowable error range; read in new data, and calculate new slope upper and lower limits in combination with the slope limit rate interval; compare the slope upper and lower limits of the new data with the current slope upper and lower limits, and decide whether to save the new data. The present invention aims at the problems that the industrial Internet of Things real-time system generates a large amount of data in a short period of time, there is a lot of data redundancy, and the real-time requirements are very high, and the compression ratio and restoration accuracy of the current lossless compression algorithm still have room for improvement. Through dynamic correction and prediction, the real-time data is quickly and efficiently lossy compressed, which can meet the system's requirements for restoration accuracy while improving the compression ratio, and can be applied to the storage requirements of real-time databases in the industrial Internet of Things.
[0068] like Figure 1As shown, a lossy compression method in an industrial Internet of Things real-time database, the steps are as follows:
[0069] Step 1: Initialize the system, read in k real-time data, and normalize the data. According to the first k data and the allowable error range, calculate the upper and lower limits of the slope and the slope limit rate. Write the data of the slope limit rate point that is less than the slope limit rate into the database, and record the slope limit rate point data as the "last written data", and the kth data as the "last read data". Initialize each correction value to 0;
[0070] Step 2: Read in new data, and use the new data, "last written data" and the error tolerance range, combined with the slope limit rate interval, to calculate the upper and lower limits of the slope;
[0071] Step 3: Compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope, and decide whether to save the new data and update the slope limit rate;
[0072] Step 4: Adjust the current slope upper and lower limits and slope correction value, and repeat step 2.
[0073] The initialization process in step 1 is: read in k real-time data P1, P2, ..., P k , and normalize it, and define the normalization formula of the i-th (i∈k) real-time collected data as follows:
[0074]
[0075] Where P i is the i-th real-time data collection, p i is the normalized data, P min is the minimum value of the data, P max is the maximum value of the data.
[0076] like Figure 2 As shown, after the data is normalized, it is sorted from small to large according to the horizontal axis to obtain the processed data: p1(t1,v1), p2(t2,v2), ..., p k (t k ,v k ), where t1, t2, … t k Indicates the time value of the collected data, v1, v2, …v k Indicates the data value of the collected data and calculates the initial slope upper limit. The steps are as follows:
[0077] S11. Traverse the data and find the slope limit two by two. Taking p1 and p2 as examples, the upper limit of the slope is: r up1 =(v2+T-v1) / (t2-t1), the lower limit of the slope is: r down1=(v2-T-v1) / (t2-t1), the slope midline is: r1=(v2-v1) / (t2-t1), we get The slope upper limit, slope lower limit and slope center line are calculated, where T is the error tolerance range ±T, and the slope correction value d is initialized. r is 0.
[0078] S12, the result obtained in S11 Sort the upper and lower limits of the slope from small to large, and take the average of the upper limits. is the upper limit of the slope r up , the lower limit average is the lower limit of the slope r down .
[0079] S13, the result obtained in S11 The slope midlines are sorted from small to large, and the initial slope limit rate α is defined at 80%. p , the 80% slope midline is defined as the initial slope limit rate α p , take the initial slope limit rate α p The corresponding data is used as the initial slope limit rate point p α .
[0080] S14, p α Recorded as "last written data" as p last (t last ,v last ), p k Recorded as "last read data" as p read (t read ,v read ).
[0081] The process of calculating the upper and lower limits of the slope described in step 2 includes:
[0082] S21, read in new data P k+1 , and normalize it to get p k+1 (t k+1 ,v k+1 ).
[0083] S22. Calculate the upper limit of the slope: r' up1 =(v k+1 +Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last ), slope lower limit: r' down1 =(v k+1 -Tv last ) / (tk+1 -t last )+d r *(t k+1 -t last ).
[0084] S23. Calculate p k+1 (t k+1 ,v k+1 ) point slope limit rate interval: Set the adjustment interval to λ(1≤λ≤n, n is p k+1 With p α The number of interval data), calculate the adjusted slope limit rate: α p ±λσ (the initial σ is 0.05), and sort the values from small to large to form the slope limit rate interval of the point, taking the maximum and minimum values r' up2 、r' down2 .
[0085] S24, take r' up =min{r' up1 ,r' up2}, r' down =max{r' down1 ,r' down2}.
[0086] At the same time, due to the continuous and long-term operation of the monitoring point data collection, the system deployment working status is regularly diagnosed and updated. Based on the calculation mode determined for the first time, the value of σ will be regularly adjusted for the collection system of the monitoring point. The process of the method for regularly diagnosing and updating the working status of the monitoring point is as follows: Figure 3 As shown. Each time the size of σ is adjusted, the corresponding slope limit rate and interval will be calculated and updated. The new slope limit rate and interval are verified by re-collecting actual data. Figure 2 When the calculation mode of the model in the calculation mode matches the best, the σ value is adjusted, otherwise the σ value remains at the last set value. According to the effect of the compressed data, adjust and update the better value, such as adjusting once a cycle. The specific adjustment strategy and method should be selected in combination with the actual application system. The regular adjustment process of the monitoring point is shown in Table 1. The present invention can ensure that the system is in a reliable and efficient operation state for a long time by regularly updating and confirming the σ value and the slope limit rate and interval.
[0087] Table 1 Adjustment strategy
[0088]
[0089]
[0090] The comparison process described in step 3 is as follows: If r' up >rup And r' down >r up or r' down <r down And r' up <r down , then new data needs to be saved, and the slope limit rate point is updated to P k+1 , and update the slope limit rate α p , otherwise, the new data is not saved and only the current slope is readjusted.
[0091] The adjustment process described in step 4 is as follows:
[0092] S41. According to the result of step 3, if new data needs to be saved, calculate the slope correction value:
[0093] d r =((v k+1 -v read ) / (t k+1 -t read )-r) / (t read -t last ).
[0094] S42, draw a vertical line through the "last read data", if r' up >r up , and the vertical line is connected to r up The intersection is written into the database and recorded as the "last written data". If r' down <r down , and the vertical line is connected to r down The intersection point is written into the database and recorded as the "last written data", and the upper and lower limits are recalculated:
[0095] r up =(v k+1 +Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last );
[0096] r down =(v k+1 -Tv last ) / (t k+1 -t last )+d r *(t k+1 -t last );
[0097] S43, if there is no need to save new data, if r' up <r up, adjust r up for r' up , if r' down >r down , adjust r down for r' down .
[0098] S44. Regardless of whether new data is saved, the current data is recorded as the “last read data”.
[0099] The decompression process is as follows Figure 4 As shown, the following steps are included: suppose the query time is t, then read the three most recent data before t p1(t1,v1), p2(t2,v2), p3(t3,v3), calculate p t (t,v t ) is as follows: k1 = (v2-v1) / (t2-t1), k2 = (v3-v2) / (t3-t2), v t =v3+(t3-t2)(k2+(k2-k1)(t3-t2) / (t2-t1)).
[0100] Example 2
[0101] like Figure 5 As shown, this embodiment provides a single-point lossy compression device for an industrial Internet of Things real-time database, including:
[0102] A data acquisition module is used to collect data based on a preset time period and perform data preprocessing on the collected data;
[0103] The data initialization module is used to calculate the slopes between the first k pre-processed data, obtain the upper and lower limits of the current slope based on the preset error tolerance range and sort them, and obtain the current slope limit rate, determine the data corresponding to the slope limit rate as the slope limit rate point, write the data before the slope limit rate point into the database, and record the slope limit rate point as the "last written data", and the kth data as the "last read data";
[0104] The data calculation module is used to read in new data, calculate a new slope for the new data and the "last written data" based on a preset initial correction value; adjust the current slope limit rate at the same time to obtain a slope limit rate interval for the new data; and obtain the upper and lower limits of the slope of the new data by combining the new slope and the slope limit rate interval;
[0105] The data comparison and saving module is used to compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope. If the upper and lower limits of the slope of the new data are both greater than the upper limit of the current slope or the upper and lower limits of the slope of the new data are both less than the lower limit of the current slope, the new data is saved and the new data is used as the updated slope limit rate point, and the data of the updated slope limit rate point is recorded as the "last read data";
[0106] The data update module is used to calculate the slope limit rate corresponding to the updated slope limit rate point as the updated current slope limit rate; at the same time, the slope correction value is calculated, and the slope upper and lower limits are calculated based on the slope correction value as the updated current slope upper and lower limits.
[0107] The data update module is also used to readjust the current upper and lower limits of the slope when new data is not saved.
[0108] This system consists of a terminal layer, an edge layer, and a platform layer. Data collection and compression processing are deployed in the edge layer, and the interactive work of model calculation and update is deployed on the edge side close to the terminal, which can reduce the delay of remote transmission of system data and improve the working performance of the system. During monitoring, IntAgent adopts a periodic model update method, that is, after the set model update cycle arrives, the model update calculation process will be automatically executed. The cycle time can be configured by the service platform of the platform layer, and the cycle interval can be adjusted. An intelligent agent (i.e., IntAgent) software is implanted in each terminal. The IntAgent is responsible for receiving the data compression work strategy sent by the platform layer (such as IntAgent's model parameters, model update cycle, etc.), and completing the model update and calculation processing of data compression according to the compression strategy.
[0109] Each edge node in the edge layer performs data collection, model calculation and update, and performs compression operations based on the model after parameter update; the edge nodes adopt a regionalized (different areas within the factory or multiple factory areas) deployment scheme, and distributed processing of edge data collection and compression. Platform layer: collection scheme definition, compression parameter setting, diagnosis prediction model; the collection scheme definition refers to the association between the collection point and the edge node (EN); the compression parameter setting is used to set the collection cycle, σ parameter, etc.; the diagnosis prediction model identifies and self-analyzes the abnormal collection data on site, and performs predictions through continuous sampling data.
[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.
Claims
1. A single-point lossy compression method for an industrial Internet of Things real-time database, characterized in that: The steps include: Collect data based on a preset time period and perform data preprocessing on the collected data; Calculate the slopes between the first k preprocessed data, obtain the upper and lower limits of the current slope based on the preset error tolerance range and sort them, and obtain the current slope limit rate, determine the data corresponding to the slope limit rate as the slope limit rate point, write the data before the slope limit rate point into the database, and record the slope limit rate point as the "last written data", and the kth data as the "last read data"; Read in new data, calculate new slope for new data and "last written data" based on preset initial correction value; adjust current slope limit rate to obtain slope limit rate interval of new data; combine new slope and slope limit rate interval to obtain upper and lower limits of slope of new data; Compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope. If the upper and lower limits of the slope of the new data are both greater than the upper limit of the current slope or the upper and lower limits of the slope of the new data are both less than the lower limit of the current slope, save the new data and use the new data as the updated slope limit rate point. Record the data of the updated slope limit rate point as "the last read data"; Calculate the slope limit rate corresponding to the updated slope limit rate point as the updated current slope limit rate; calculate the slope correction value at the same time, and calculate the slope upper and lower limits based on the slope correction value as the updated current slope upper and lower limits; Read in new data again and repeat the calculation and judgment steps until all data compression is completed.
2. According to claim 1, a single-point lossy compression method for an industrial Internet of Things real-time database is characterized in that: The data preprocessing process includes: reading k real-time data P1, P2, ..., P k , and normalize it, and define the normalization formula of the i-th (i∈k) real-time collected data as follows: Where P i is the i-th real-time data collection, p i is the normalized data, P min is the minimum value of the data, P max is the maximum value of the data. After the data is normalized, it is arranged in time series to obtain the processed data: p1(t1,v1), p2(t2,v2), ..., p k (t k ,v k ), where t1, t2, … t k Indicates the time value of the collected data, v1, v2, …v k The data value representing the collected data.
3. According to claim 1, a single-point lossy compression method for an industrial Internet of Things real-time database is characterized in that: The process of calculating the slope of the first k initial data includes: S11, traverse the data, based on the preset error tolerance range, calculate the slope limit of the data pairwise, and obtain The upper slope limit, The lower slope limit and The slope of the midline; S12, the result obtained in S11 The upper and lower limits of the slopes are sorted from small to large, and the average of the upper and lower limits of the slopes is taken as the upper limit of the slope, and the average of the lower limits of the slopes is taken as the lower limit of the slope; S13, the result obtained in S11 The slope midlines are sorted from small to large, and the initial slope limit rate is selected from the sorted slope midlines according to a preset ratio, and the data corresponding to the initial slope limit rate is taken as the initial slope limit rate point; S14, the initial slope limit rate point is recorded as the "last written data", and the kth data is recorded as the "last read data".
4. The single-point lossy compression method for an industrial Internet of Things real-time database according to claim 1 is characterized in that: The process of calculating the new slope using the new data and the "last written data" includes: S21, read in new data and perform normalization processing; S22, calculating the slope limit of the new data according to the "last written data", calculating a new upper slope limit, and calculating a new lower slope limit; S23, set the adjustment interval to λ, and calculate the adjusted slope limit rate: α p ±λσ, and sort them from small to large to form the slope limit rate interval of the point, taking the maximum and minimum values of the interval, where λ is the adjustment interval; S24. The smaller of the new upper slope limit and the interval maximum value is taken as the upper slope limit of the new data, and the smaller of the new lower slope limit and the interval minimum value is taken as the lower slope limit of the new data.
5. The single-point lossy compression method for an industrial Internet of Things real-time database according to claim 1 is characterized in that: Compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope. If either of the upper and lower limits of the slope of the new data is not greater than the upper limit of the current slope or either of the upper and lower limits of the slope of the new data is not less than the lower limit of the current slope, determine not to save the new data.
6. The single-point lossy compression method for an industrial Internet of Things real-time database according to claim 5 is characterized in that: If the new data is not saved, the current upper and lower limits of the slope will be readjusted. The specific rules for adjusting the current upper and lower limits of the slope are as follows: if the upper limit of the slope of the new data is less than the upper limit of the current slope, the upper limit of the slope of the new data is adjusted as the upper limit of the current slope after the update; if the lower limit of the slope of the new data is greater than the lower limit of the current slope, the lower limit of the slope of the new data is adjusted as the lower limit of the current slope after the update.
7. The single-point lossy compression method for an industrial Internet of Things real-time database according to claim 1 is characterized in that: If new data is saved, the slope midline of the slope limit rate point is calculated as the current slope limit rate after update.
8. The single-point lossy compression method for an industrial Internet of Things real-time database according to claim 1 is characterized in that: Regardless of whether the new data is saved or not, the new data is recorded as the "last read data".
9. A single-point lossy compression device for an industrial Internet of Things real-time database, used to implement the method described in any one of claims 1 to 8, characterized in that: include: A data acquisition module is used to collect data based on a preset time period and perform data preprocessing on the collected data; The data initialization module is used to calculate the slopes between the first k preprocessed data, obtain the upper and lower limits of the current slope based on the preset error tolerance range and sort them, and obtain the current slope limit rate, determine the data corresponding to the slope limit rate as the slope limit rate point, write the data before the slope limit rate point into the database, and record the slope limit rate point as the "last written data", and the kth data as the "last read data"; The data calculation module is used to read in new data, calculate a new slope for the new data and the "last written data" based on a preset initial correction value; adjust the current slope limit rate to obtain a slope limit rate interval for the new data; and combine the new slope and the slope limit rate interval to obtain the upper and lower limits of the slope of the new data; The data comparison and saving module is used to compare the upper and lower limits of the slope of the new data with the upper and lower limits of the current slope. If the upper and lower limits of the slope of the new data are both greater than the upper limit of the current slope or the upper and lower limits of the slope of the new data are both less than the lower limit of the current slope, the new data is saved and the new data is used as the updated slope limit rate point, and the data of the updated slope limit rate point is recorded as "the last read data"; The data update module is used to calculate the slope limit rate corresponding to the updated slope limit rate point as the updated current slope limit rate; at the same time, the slope correction value is calculated, and the slope upper and lower limits are calculated based on the slope correction value as the updated current slope upper and lower limits.
10. The single-point lossy compression device for an industrial Internet of Things real-time database according to claim 9, characterized in that: The data update module is also used to readjust the current upper and lower limits of the slope when new data is not saved.
Citation Information
Patent Citations
Compression storage method for time sequence data, electronic equipment and storage medium
CN108197181A
Methods for lossy compression and decompression in real-time database through dynamic prediction
CN103136202A
Data compression device, data compression method, and computer program product
CN104160629A
Compression method suitable for agricultural sensing data
CN108540136A
Data compression method for industrial real-time database
CN112667582A
