Information compression and storage methods for large-scale data analysis

Abstract

This invention relates to the field of data compression technology, specifically to an information compression and storage method for large-scale data analysis. The method includes: acquiring real-time data segments and historical datasets; designating the first data point of the real-time data segment as the initial data point; determining the anomaly level of the initial data point based on the differences in data values ​​between the real-time data segment and the historical dataset, and obtaining an initial threshold value; acquiring the first compressed segment and its outliers based on the initial threshold value; acquiring the threshold value of the next compressed segment based on the first compressed segment and its outliers; thereby obtaining the threshold value and outliers for each compressed segment; and completing the compression and storage of real-time data based on the threshold value and outliers of each compressed segment. This invention improves the compression performance of the rotating door compression algorithm.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and more specifically to a method for compressing and storing information for large-scale data analysis. Background Technology

[0002] SCADA (Supervisory Control and Data Acquisition) systems are computer-based control systems used for real-time monitoring and control of industrial processes, widely applied in power, petroleum, transportation, and other fields. For example, in the power sector, various power parameters (such as voltage, current, and frequency) need to be collected at high frequency, covering countless monitoring points (such as substations, lines, and intelligent devices) across a vast geographical area. This results in the generation of massive real-time data streams per second, requiring extremely high data sampling rates and accuracy. Furthermore, the need for long-term storage of historical data (for trend analysis, auditing, and compliance) necessitates the complete archiving of data for many years, consuming enormous storage resources. Therefore, it is necessary to compress and store the collected data to reduce disk I / O overhead, save compression space, and improve data processing speed.

[0003] Traditional data compression methods are generally divided into lossless compression and lossy compression. Lossless compression restores the original data after decompression, but the compression ratio is limited. Lossy compression results in some data loss after decompression, but the compression ratio is usually higher. In SCADA system data compression technology, the rotating door compression algorithm is a classic lossy compression algorithm. However, the choice of gate compression algorithm is highly dependent on the selection of the threshold value. In addition, the rotating door compression algorithm processes all data points in the same way, which cannot handle outliers. The presence of many outliers in the data will greatly affect the compression ratio of the rotating door compression algorithm. Summary of the Invention

[0004] To address the technical problem that the compression ratio of the rotating door algorithm is significantly affected, this invention provides an information compression and storage method for large-scale data analysis. The specific technical solution adopted is as follows:

[0005] This invention proposes an information compression and storage method for large-scale data analysis, which includes the following steps:

[0006] Obtain real-time data segments and historical datasets;

[0007] The first data point of the real-time data segment is recorded as the initial data point. The degree of anomaly of the initial data point is obtained based on the difference between the data value of the data point in the real-time data segment and the data value in the historical dataset, as well as the number of data points in the real-time data segment. An initial threshold value is obtained based on the degree of anomaly of the initial point.

[0008] The real-time data segment is compressed for the first time based on the initial threshold value to obtain the first compressed segment and its outliers. The threshold value of the second compressed segment is obtained based on the first compressed segment and its outliers. The threshold value of the second compressed segment and its outliers is obtained based on the second compressed segment and its outliers. The threshold value of the third compressed segment is obtained based on the second compressed segment and its outliers, and so on. The threshold value of the next compressed segment is obtained based on the threshold value and outliers of the previous compressed segment.

[0009] Real-time data compression and storage are completed based on the threshold value of each compression segment and its outliers.

[0010] Preferably, the method for obtaining real-time data segments and historical datasets is as follows:

[0011] For each type of data, data is collected once every 1 second for a preset time period. The collected data is recorded as a real-time data segment. The real-time data segment is collected up to the current time, and all data before the real-time data segment is recorded as a historical dataset.

[0012] Preferably, the method for determining the degree of anomaly of the initial data point based on the difference between the data value of the data point in the real-time data segment and the data value in the historical dataset, as well as the number of data points in the real-time data segment, is as follows:

[0013] A data interval is formed by the data values ​​of all data points in the historical dataset, which is the interval from the minimum data value to the maximum data value in the historical dataset. Data points in the real-time data segment whose data values ​​fall within the data interval are extracted and recorded as interval data points. The degree of anomaly of the initial data points is obtained based on the ratio of the interval data points to all data points in the real-time data segment and the difference between the data values ​​of the data points in the real-time data segment and the maximum and minimum data values ​​of the historical dataset.

[0014] Preferably, the method for obtaining the anomaly degree of the initial data point based on the ratio of the interval data point to all data points in the real-time data segment and the difference between the data value of the data point in the real-time data segment and the maximum and minimum data values ​​of the historical dataset is as follows:

[0015]

[0016] In the formula, This represents the minimum data value in the historical dataset. This represents the maximum data value in the historical dataset. This represents the data value of the i-th data point in the real-time data points. This indicates the number of data points in the real-time data segment. This indicates the number of data points in the interval of the real-time data segment. This represents an exponential function with the natural constant as its base. This indicates the degree of anomaly in the initial data points.

[0017] Preferably, the method for obtaining the initial threshold value based on the degree of anomaly at the initial point is as follows:

[0018] The product of the anomaly level of the initial data point and the preset compression threshold is used as the initial threshold value of the real-time compression segment.

[0019] Preferably, the method for obtaining the first compressed segment and its outliers by performing the first compression on the real-time data segment based on the initial threshold value is as follows:

[0020] The point obtained by adding an initial threshold value to the ordinate of the initial data point in the real-time data segment is denoted as the upper pivot point. The point obtained by subtracting the initial threshold value from the ordinate of the initial data point in the real-time data segment is denoted as the lower pivot point. Connect the upper pivot point to the next data point to obtain a line and calculate its slope, which is denoted as the first slope. Connect the lower pivot point to the next data point to obtain a line and calculate its slope, which is denoted as the second slope. Let the sum of the two angles between the two lines be the included angle between the two lines. When the included angle between the two lines corresponding to the first slope and the second slope is less than 180 degrees, connect the upper pivot point to the next data point after that to obtain a line and calculate its slope. Save the slope that is larger between the newly calculated slope and the first slope and denoted as the new first slope. Connect the lower pivot point to the next data point after that to obtain a line and calculate its slope. Save the slope that is smaller between the newly calculated slope and the second slope and denoted as the new second slope. Based on the continuous updating of the first slope and the second slope, obtain the first compressed segment and its corresponding outlier point.

[0021] Preferably, the method for obtaining the first compressed segment and its corresponding outlier point based on the continuous updating of the first slope and the second slope is as follows:

[0022] If the angle between the two lines corresponding to the first slope and the second slope is less than 180 degrees, update the first slope and the second slope again through the next data point until the angle between the two lines corresponding to the first slope and the second slope is greater than or equal to 180 degrees. Mark the data point at this time as a pending data point. Do not update the first slope and the second slope at this time. Then continue to update the first slope and the second slope for the next data point. If the angle between the two lines is still greater than or equal to 180 degrees, stop the operation and set the last data point with an angle between the two lines less than 180 degrees as the end point of the compressed segment. If the angle between the two lines is less than 180 degrees, mark the pending data point as an outlier and continue to traverse the next data point to update the first slope and the second slope until there are two consecutive data points with updated first slopes and second slopes that are both greater than or equal to 180 degrees. Obtain the end point of the compressed segment. The data points between the initial data point and the end point are the first compressed segment.

[0023] Preferably, the method for obtaining the threshold value of the next compression segment based on the threshold value of the previous compression segment and the outlier is as follows:

[0024] The previous compression segment is designated as the known compression segment. The line connecting the first slope of the known compression segment is designated as the upper pivot line, and the line connecting the second slope is designated as the lower pivot line. The Euclidean distances from the outliers of the known compression segment to the two pivot lines are calculated. The smallest Euclidean distance is designated as the outlier distance of the outlier. The step size between the outliers of the known compression segment is obtained, which is the number of data points between the outliers plus one. The threshold value of the next compression segment is obtained based on the outlier distance, the step size between the outliers, the number of outliers, and the threshold value of the known compression segment.

[0025] Preferably, the method for obtaining the threshold value of the next compression segment based on the outlier distance, the step size between outliers, the number of outliers, and the known threshold value of the compression segment is as follows:

[0026]

[0027] In the formula, This represents the threshold value of a known compressed segment. This represents the outlier distance of the j-th outlier point in the c-th compressed segment. This represents the number of outliers in the c-th compressed segment. Indicates hyperparameters, This represents the step size between the j-th outlier in the c-th compressed segment and the next outlier. This represents the number of data points in the c-th compressed segment. This represents an exponential function with the natural constant as its base. This represents the threshold value for the next compression segment of a known compression segment.

[0028] Preferably, the method for compressing and storing real-time data based on the threshold value and outlier points of each compression segment is as follows:

[0029] The straight line data from the start point to the end point of each compression segment is used as the first compression part, and the outlier points corresponding to each compression segment are used as the second compression part. The two compression parts are retained to complete the compression and storage of real-time data.

[0030] The present invention has the following beneficial effects: The present invention proposes an information compression and storage method for large-scale data analysis. By analyzing the data, outliers in the data are obtained. An initial compression segment is obtained by analyzing the real-time data as a whole. The next compression segment is obtained based on the initial compression segment and its outliers. The next compression segment is obtained by sequentially using the previous compression segment and its outliers. In this way, the threshold value is adaptively adjusted according to the changing trend of the data, which greatly improves the compression performance of the rotating door compression algorithm without increasing the algorithm complexity. Attached Figure Description

[0031] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of an information compression and storage method for large-scale data analysis provided in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of outliers;

[0034] Figure 3 This is a schematic diagram of a traditional revolving door compression algorithm;

[0035] Figure 4 This is a schematic diagram of the improved rotating door compression algorithm and its outliers. Detailed Implementation

[0036] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the information compression and storage method for large-scale data analysis proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] Example of an information compression and storage method for large-scale data analysis:

[0039] The specific solution of the information compression and storage method for large-scale data analysis provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0040] Please see Figure 1 The diagram illustrates a flowchart of an information compression and storage method for large-scale data analysis according to an embodiment of the present invention. The method includes the following steps:

[0041] Step S001: Obtain real-time data segments and historical datasets.

[0042] Production equipment data typically consists of vibration data collected by sensors, and data from various sensors such as current sensors and voltage sensors. Each type of sensor data is categorized as a data type. In this embodiment, we describe each data type individually. For each data type, a segment is collected every 10 minutes, with each collection session lasting 10 minutes. Data is collected every 1 second. This collected data is designated as a real-time data segment. Since real-time data may be used for prediction and other purposes, it is transmitted to a real-time database. Real-time data segments are usually critical data with a small volume and are not compressed in the real-time database. Data prior to the real-time data segment is stored in a historical database. When the real-time data segment is no longer needed in the real-time database, the real-time database pushes a message to the historical database via a message push mechanism, transferring the real-time data segment to the historical database for long-term storage. At this point, the real-time data segment needs to be compressed. Each data type supports functions such as historical alarm archiving, historical curve viewing, and online or offline learning for intelligent alarms. The data in the historical database before compression is designated as historical data, and the collection of all historical data is designated as the historical dataset.

[0043] At this point, the real-time data segment and the historical dataset have been obtained.

[0044] Step S002: Record the first data point of the real-time data segment as the initial data point; obtain the degree of anomaly of the initial data point based on the difference between the data value of the data point in the real-time data segment and the data value in the historical dataset, as well as the number of data points in the real-time data segment; obtain the initial threshold value based on the degree of anomaly of the initial point.

[0045] The rotating door algorithm is used to compress real-time data segments. The threshold value of rotating door compression determines the compression ratio and compression accuracy. A larger threshold value includes more data points, meaning more data points are discarded, increasing the compression ratio, but also increasing the compression error. Conversely, a smaller threshold value includes fewer data points, meaning fewer data points are discarded, decreasing the compression ratio, but also decreasing the compression error. Traditional rotating door algorithms use manually assigned threshold values ​​that cannot be changed during the compression process, resulting in poor controllability and failing to consider the specific characteristics of data at different times. Therefore, this embodiment obtains a suitable initial threshold value to improve compression accuracy.

[0046] For the first data point in the real-time data segment, use it as the reference point. Use the two points above and below the reference point's threshold as the fulcrums of a rotating door. Connect the second data point to these two fulcrums and calculate their slopes. Then connect the third data point to these two fulcrums and calculate their slopes. Designate the fulcrum above the reference point as the upper fulcrum and the fulcrum below the reference point as the lower fulcrum. Compare the slopes of the upper fulcrum with those of different data points and save the line with the larger slope. Compare the slopes of the lower fulcrum with those of different data points and save the line with the smaller slope. Then connect each data point to a fulcrum, calculate the slope, and compare and save the lines. Continue this process until the angle between the line connecting the upper and lower fulcrums is greater than or equal to 180 degrees. Stop the operation, store the previous data point, and start a new compression segment from that point.

[0047] In this embodiment, the collected data is real-time data, and the amount of collected data is usually small. The first data point of the real-time data segment is recorded as the initial data point. For each type of data, the data format of the initial data point is different. Therefore, it is necessary to adaptively obtain the initial threshold value according to the degree of anomaly of the initial data point of the real-time data segment. The maximum and minimum data values ​​in the historical dataset constitute the data interval, denoted as... The degree of anomaly of the initial data points is determined based on the maximum and minimum values ​​in the historical dataset, the data value of each data point in the real-time data segment, and the number of data points within the data interval. The formula is as follows:

[0048]

[0049] In the formula, This represents the minimum data value in the historical dataset. This represents the maximum data value in the historical dataset. This represents the data value of the i-th data point in the real-time data points. This indicates the number of data points in the real-time data segment. This indicates the number of data points in the data range whose data values ​​are located within the real-time data segment. This represents an exponential function with the natural constant as its base. This indicates the degree of anomaly in the initial data points.

[0050] The greater the fluctuation in real-time data points, the faster and more pronounced the data change trend, and the more likely the equipment is to be malfunctioning. Simultaneously, the greater the amount of data deviating from the normal operating range, the more likely the equipment is in an abnormal operating state. Therefore, the real-time data segment is of greater importance, as it can reflect abnormal equipment operation. In this case, the data loss rate should be as small as possible to facilitate anomaly analysis. Thus, the threshold value should be relatively small to ensure data compression accuracy. The initial threshold value is obtained from this, using the following formula:

[0051]

[0052] In the formula, This represents the threshold value for human experience compression during historical compression. Indicates the degree of anomaly of the initial data points. This represents the initial threshold value for the first compressed segment.

[0053] At this point, the initial threshold value has been obtained.

[0054] Step S003: Perform the first compression on the real-time data segment based on the initial threshold value to obtain the first compressed segment and its outliers. Obtain the threshold value of the second compressed segment based on the first compressed segment and its outliers. Obtain the threshold value of the second compressed segment and its outliers based on the threshold value of the second compressed segment. Obtain the threshold value of the third compressed segment based on the second compressed segment and its outliers, and so on. Obtain the threshold value of the next compressed segment based on the threshold value and outliers of the previous compressed segment.

[0055] The initial threshold value is the threshold value corresponding to the initial data points. In subsequent processing, the distribution and changes of the data will differ from the initial time. If the same threshold value is used globally, the compression accuracy and compression performance will be poor, and there will also be outliers in the data. Outliers will affect the compression performance of the preceding and following segments. Therefore, the threshold value should be adjusted adaptively according to the changes in the data. In order to balance compression accuracy and compression performance, it is necessary to analyze the data, obtain outliers in the data, and adaptively adjust the threshold value according to the changing trend of the data to adjust the compression accuracy.

[0056] Given a fixed initial threshold, outliers affect compression performance, such as... Figure 2 As shown, the 6th data point in the horizontal axis sequence is the outlier. This is because the 7th and 8th data points after the 6th data point are also within the initial threshold value. If a traditional rotating door compression algorithm is used for compression at this point, such as... Figure 3 As shown, the compressed segment 1 after compression consists of arrive The straight line replaces Data points, compressed segment 2 consists of arrive The straight line replaces Data points, compressed segment 3 consists of arrive replace with a straight line Data points, compressed segment 4 consists of arrive The straight line replaces Data points, among which This represents the (o+1)th data point; in this case, only the first data point is retained after compression. The 5th data point The 6th data point The 9th data point The 15th data point .

[0057] exist Figure 2 In the above, the 6th and 9th outliers are key factors affecting compression performance. If these outliers are marked and recorded separately, such as... Figure 4 As shown, the compressed data becomes compressed segment 1. arrive The straight line replaces Data points, compressed segment 2 consists of arrive The straight line replaces The data point, that is, the first data point retained after compression. The 10th data point The 15th data point .

[0058] Therefore, outliers are identified, and the threshold value of the next compression segment is adaptively adjusted based on the outlier degree and density of the outliers, as follows:

[0059] Based on the initial threshold value The upper and lower threshold values ​​of the distance from the reference point are Using the two points as fulcrums of a revolving door, two virtual doors are established. Following the traditional revolving door rules, the termination point of the current compression segment is found. This termination point is the data point preceding the data point whose angle between the lines connecting the two fulcrums is greater than 180 degrees. Figure 3 Zhongru As shown, after determining the termination point, the second data point after the termination point is also connected to the fulcrum, and the maximum slope of the upper fulcrum and the minimum slope of the lower fulcrum are obtained respectively. If the angle between the two connecting lines is less than 180 degrees, the data point is considered to be within the compression segment as well. The data point after the termination point is recorded as an outlier. Figure 3 middle This is considered an outlier. If the angle between the two connecting lines is greater than 180 degrees, meaning neither of the two data points after the termination point is within the compressed segment, then the termination point is considered the termination point of the compressed segment, thus obtaining the first compressed segment. Figure 4 As shown, the 10th data point is... It is the end point of the compressed segment, and the two data points that follow are not in the first compressed segment.

[0060] At this time This refers to the end point of the first compression segment and the starting point of the next compression segment. There are also outliers in the first compression segment. Outliers are highly likely to be abnormalities in the production equipment. The threshold value of the next compression segment is adaptively adjusted according to the degree and density of outliers.

[0061] For the previous compression segment, the final connection between its two pivots is denoted as the upper pivot connection and the lower pivot connection, respectively. The equations of these two pivot connections are obtained, and the distances from the outliers to these two pivot connections are calculated. The minimum distance from the outlier to these pivot connections is recorded as the outlier distance. Based on the outlier distances, the initial threshold value, and the step size of the x-coordinates between the outliers, the threshold value for the next compression segment is obtained, as shown in the following formula:

[0062]

[0063] In the formula, This represents the threshold value of the c-th compressed segment. This represents the outlier distance of the j-th outlier point in the c-th compressed segment. This represents the number of outliers in the c-th compressed segment. This represents a hyperparameter; in this embodiment, the empirical value is 0.1. This represents the step size between the j-th outlier in the c-th compressed segment and the next outlier. This represents the number of data points in the c-th compressed segment. This represents an exponential function with the natural constant as its base. This represents the threshold value of the (c+1)th compressed segment.

[0064] In the above calculation, outliers belong to the data points in the compressed segment, and the step size represents the number of data points between two data points plus 1, i.e., as shown below. Figure 4 As shown, the 6th and 9th data points are discrete points, so the step size between them is 9-6=3. If there is only one discrete point in the compression segment, the last term in the formula is 1. If there is no discrete point in the compression segment, no adjustment is made, and the threshold value of the previous compression segment is used.

[0065] In the previous compression segment, the larger the deviation of outliers from the threshold value and the denser the distribution of outliers, the higher the probability of production equipment malfunction. Therefore, the accuracy requirement for subsequent data is higher, and the corresponding threshold value needs to be reduced. Similarly, the smaller the deviation of marked outliers from the threshold value and the sparser the distribution of outliers in the previous compression segment, the lower the probability of production equipment malfunction. Therefore, the accuracy requirement for subsequent data is lower, and the corresponding threshold value needs to be increased.

[0066] At this point, the threshold value of each compressed segment before compression has been obtained.

[0067] Step S004: Compress and store real-time data according to the threshold value of each compression segment and its outliers.

[0068] The data is compressed and stored using the above rules. The compressed data is divided into two parts: one part is the straight line data from the start point to the end point in the corresponding compressed segment, and the other part is the marked outlier data in the corresponding compressed segment. By compressing and storing real-time data, the compressed data is used as data support for functions such as historical alarm archiving and retrieval, historical curve viewing, and intelligent alarm online or offline learning, thereby monitoring, warning, and adaptively controlling the operation of production equipment.

[0069] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for information compression storage for large scale data analysis, characterized in that, The method comprises the following steps: acquiring a real-time data segment and a historical data set; taking a first data point of the real-time data segment as an initial data point, acquiring an abnormality degree of the initial data point according to a difference between data values of the data points in the real-time data segment and data values in the historical data set and a number of data points in the real-time data segment, and acquiring an initial threshold value according to the abnormality degree of the initial data point; according to the initial threshold value, performing first compression on the real-time data segment to acquire a first compressed segment and outliers thereof, acquiring a threshold value of a second compressed segment according to the first compressed segment and the outliers thereof, acquiring the second compressed segment and outliers thereof according to the threshold value of the second compressed segment, acquiring a threshold value of a third compressed segment according to the second compressed segment and the outliers thereof, and acquiring a threshold value of a next compressed segment according to a threshold value and outliers of a previous compressed segment; according to the threshold value and the outliers of each compressed segment, completing compressed storage of the real-time data; the method for acquiring the first compressed segment and the outliers thereof according to the initial threshold value comprises the following steps: taking a point obtained by adding the initial threshold value to the ordinate of the initial data point of the real-time data segment as an upper branch point, taking a point obtained by subtracting the initial threshold value from the ordinate of the initial data point of the real-time data segment as a lower branch point, connecting the upper branch point and a next data point to obtain a line and calculating a slope of the line as a first slope, connecting the lower branch point and the next data point to obtain a line and calculating a slope of the line as a second slope, taking two angles between the two lines as included angles of the two lines, when the included angles of the two lines corresponding to the first slope and the second slope are less than 180 degrees, connecting the upper branch point and a next data point to obtain a line and calculating a slope of the line, taking a maximum slope of the newly calculated slope and the first slope as a new first slope, connecting the lower branch point and a next data point to obtain a line and calculating a slope of the line, taking a minimum slope of the newly calculated slope and the second slope as a new second slope, and acquiring the first compressed segment and corresponding outliers thereof according to continuous updating of the first slope and the second slope.

2. The information compression storage method for large scale data analysis of claim 1, wherein, the method for acquiring the real-time data segment and the historical data set comprises the following steps: collecting data every 1s for each type of data, collecting data for a preset time, taking the collected data as a real-time data segment, collecting data until a current time, and taking all data before the real-time data segment as a historical data set.

3. The information compression storage method for large scale data analysis of claim 1, wherein, the method for acquiring the abnormality degree of the initial data point according to a difference between data values of data points in the real-time data segment and data values in the historical data set and a number of data points in the real-time data segment comprises the following steps: constructing a data interval with data values of all data points in the historical data set, taking data points with data values in the data interval in the real-time data segment as interval data points, and acquiring the abnormality degree of the initial data point according to a ratio of the interval data points to all data points in the real-time data segment and a difference between data values of the data points in the real-time data segment and maximum and minimum data values in the historical data set.

4. The information compression storage method for large scale data analysis of claim 3, wherein, The method for obtaining the abnormal degree of the initial data point according to the ratio of the interval data point and all data points in the real-time data segment and the difference between the data value of the data point in the real-time data segment and the maximum data value and the minimum data value of the historical data set is: wherein, represents the minimum data value in the historical data set, represents the maximum data value in the historical data set, represents the data value of the i-th data point in the real-time data points, represents the number of data points in the real-time data segment, represents the number of interval data points in the real-time data segment, represents an exponential function with a natural constant as the base, represents the abnormality degree of the initial data point.

5. The information compression storage method for large scale data analysis of claim 1, wherein, The method for obtaining the initial threshold value according to the abnormal degree of the initial point is: The product of the abnormal degree of the initial data point and the preset compression threshold value is taken as the initial threshold value of the real-time compression segment.

6. The information compression storage method for large scale data analysis of claim 1, wherein, The method for obtaining the first compression segment and the corresponding outlier point according to the continuously updated first slope and second slope is: If the included angle of the two lines corresponding to the first slope and the second slope is less than 180 degrees, the first slope and the second slope are updated again through the next data point until the included angle of the two lines corresponding to the first slope and the second slope is greater than or equal to 180 degrees, the data point at this time is marked as a pending data point, the first slope and the second slope are not updated at this time, and the first slope and the second slope are continuously updated for the next data point, if the included angle of the two lines is still greater than or equal to 180 degrees at this time, the operation is stopped, and the data point of the last two lines with an included angle less than 180 degrees is taken as the termination point of the compression segment; if the included angle of the two lines is less than 180 degrees at this time, the pending data point is marked as an outlier, the first slope and the second slope are continuously updated for the next data point until the first slope and the second slope of two continuous data points are greater than or equal to 180 degrees, the termination point of the compression segment is obtained, and the data points between the initial data point and the termination point are the first compression segment.

7. The information compression storage method for large scale data analysis of claim 6, wherein, The method for obtaining the threshold value of the next compression segment according to the threshold value and the outlier of the previous compression segment is: The previous compression segment is marked as a known compression segment, the line corresponding to the first slope of the known compression segment is marked as an upper branch point line, and the line corresponding to the second slope is marked as a lower branch point line, the Euclidean distances of the outliers of the known compression segment to the two branch point lines are calculated, the minimum Euclidean distance of the outliers is marked as the outlier distance of the outliers, the step length between the outliers of the known compression segment is obtained, the step length is the number of data points between the outliers plus one, and the threshold value of the next compression segment is obtained according to the outlier distance of the outliers, the step length between the outliers, the number of outliers and the threshold value of the known compression segment.

8. The information compression storage method for large scale data analysis of claim 7, wherein, The method for obtaining the threshold value of the next compression segment according to the outlier distance of the outliers, the step length between the outliers, the number of outliers and the threshold value of the known compression segment is: wherein denotes the threshold value of the known compression segment, denotes the outlier distance of the jth outlier in the cth compression segment, denotes the number of outliers in the cth compression segment, denotes a hyperparameter, denotes the step length between the jth outlier and the next outlier in the cth compression segment, denotes the number of data points in the cth compression segment, denotes the exponential function with base of the natural constant, denotes the threshold value of the next compression segment of the known compression segment.

9. The information compression storage method for large scale data analysis of claim 1, wherein, The method for completing the compression storage of the real-time data according to the threshold value of each compression segment and the outliers thereof is: The straight line data of each compression segment from the starting point to the termination point is taken as a first compression part, the outliers corresponding to each compression segment are taken as a second compression part, and the compression storage of the real-time data is completed by retaining the two compression parts.

Images