Information compression storage method for large-scale data analysis

Abstract

The invention relates to the technical field of data compression, in particular to an information compression storage method for large-scale data analysis. The method comprises the following steps: acquiring a real-time data segment and a historical data set; recording a first data point of the real-time data segment as an initial data point, obtaining an abnormal degree of the initial data point according to a data value difference between the real-time data segment and the historical data set, and obtaining an initial threshold value; and acquiring a first compression section and an outlier thereof according to the initial threshold value, acquiring a threshold value of a next compression section according to the first compression section and the outlier thereof, thereby acquiring a threshold value and an outlier of each compression section, and completing compression storage of real-time data according to the threshold value and the outlier of each compression section. The compression performance of the revolving door compression algorithm is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression, in particular to an information compression storage method for large-scale data analysis. BACKGROUND

[0002] SCADA (Supervisory Control and Data Acquisition) system is a computer-based control system for real-time monitoring and control of industrial processes, widely used in power, oil, transportation and other fields. For example, in the field of electric power, various power parameters (such as voltage, current, frequency) need to be collected at high frequency, and cover numerous monitoring points (such as substations, lines, intelligent devices) in a wide geographical range. This results in the generation of a large amount of real-time data stream every second, with extremely high data sampling rate and precision requirements. In addition, long-term storage of historical data is required (for trend analysis, audit and compliance), and data needs to be completely archived for many years, occupying huge storage resources. At this time, the collected data needs to be compressed and stored to reduce disk I / O overhead, save compression space and improve data processing speed.

[0003] Traditional data compression methods are usually divided into lossless compression and lossy compression. The compressed data is restored to the original data after decompression, but the compression rate is limited. The compressed data after lossy compression will have some loss after decompression, but the compression rate of lossy compression is usually larger. In the SCADA system data compression technology, the rotating door compression algorithm is a classic lossy compression algorithm, but the selection of the threshold value is very dependent on the selection of the threshold value, and the rotating door compression algorithm has the same processing method for all data points. It cannot handle abnormal points, and the presence of many abnormal points in the data will greatly affect the compression rate of the rotating door compression algorithm. SUMMARY

[0004] In order to solve the technical problem that the compression rate of the rotating door algorithm is greatly affected, the present application provides an information compression storage method for large-scale data analysis, and the technical scheme adopted is as follows: The present application provides an information compression storage method for large-scale data analysis, which comprises the following steps: Obtaining a real-time data segment and a historical data set; The first data point of the real-time data segment is recorded as an initial data point, and the abnormality degree of the initial data point is obtained according to the difference between the data value of the data point in the real-time data segment and the data value in the historical data set and the number of data points in the real-time data point. The initial threshold value is obtained according to the abnormality degree of the initial point; According to the initial threshold value, the first compression segment and its outliers are obtained by compressing the real-time data segment, the threshold value of the second compression segment is obtained according to the first compression segment and its outliers, the second compression segment and its outliers are obtained according to the threshold value of the second compression segment, the threshold value of the third compression segment is obtained according to the second compression segment and its outliers, and so on, and the threshold value of the next compression segment is obtained according to the threshold value and outliers of the last compression segment. According to the threshold value of each compression segment and its outliers, the compression storage of the real-time data is completed.

[0005] Preferably, the method for obtaining the real-time data segment and the historical data set is: For each type of data, data is collected every 1s, and data of a preset time is collected, and the collected data is recorded as a real-time data segment, the real-time data segment is collected to the current time, and all data before the real-time data segment is recorded as a historical data set.

[0006] Preferably, the method for obtaining the abnormal degree of the initial data point according to the difference between the data value of the data point in the real-time data segment and the data value in the historical data set and the number of data points in the real-time data point is: The data values of all data points in the historical data set form a data interval, the data interval is an interval formed by the minimum data value to the maximum data value in the historical data set, the data points with data values in the data interval in the real-time data segment are extracted and recorded as interval data points, and the abnormal degree of the initial data point is obtained according to the ratio of the interval data points to all data points in the real-time data segment and the difference between the data value of the data point in the real-time data segment and the maximum data value and the minimum data value of the historical data set.

[0007] Preferably, the method for obtaining the abnormal degree of the initial data point according to the ratio of the interval data points to all data points in the real-time data segment and the difference between the data value of the data point in the real-time data segment and the maximum data value and the minimum data value of the historical data set is: In the formula, represents the minimum data value in the historical data set, represents the maximum data value in the historical data set, represents the data value of the i-th data point in the real-time data point, represents the number of data points in the real-time data segment, represents the number of interval data points in the real-time data segment, represents an exponential function with a natural constant as the base, represents the abnormal degree of the initial data point.

[0008] Preferably, the method for obtaining the initial threshold value according to the abnormal degree of the initial point is: The product of the abnormality degree of the initial data point and the preset compression threshold value is taken as an initial threshold value of the real-time compression segment.

[0009] Preferably, the method for obtaining the first compression segment and its outlier point by first compressing the real-time data segment according to the initial threshold value is as follows: The point obtained by adding the initial threshold value to the ordinate of the initial data point of the real-time data segment is recorded as an upper branch point, the point obtained by subtracting the initial threshold value from the ordinate of the initial data point of the real-time data segment is recorded as a lower branch point, a line is obtained by connecting the upper branch point and the next data point, and the slope of the line is calculated and recorded as a first slope; a line is obtained by connecting the lower branch point and the next data point, and the slope of the line is calculated and recorded as a second slope; the two angles between the two lines are taken as the included angles of the two lines; when the included angles of the two lines corresponding to the first slope and the second slope are less than 180 degrees, a line is obtained by connecting the upper branch point and the next data point, and the slope of the line is calculated; the maximum slope of the newly calculated slope and the first slope is taken as a new first slope; a line is obtained by connecting the lower branch point and the next data point, and the slope of the line is calculated; the minimum slope of the newly calculated slope and the second slope is taken as a new second slope; and the first compression segment and its corresponding outlier point are obtained according to the continuous updating of the first slope and the second slope.

[0010] Preferably, the method for obtaining the first compression segment and its corresponding outlier point according to the continuous updating of the first slope and the second slope is as follows: If the included angles of the two lines corresponding to the first slope and the second slope are less than 180 degrees, the first slope and the second slope are updated again through the next data point until the included angles of the two lines corresponding to the first slope and the second slope are greater than or equal to 180 degrees, the data point at this time is marked as a pending data point, the first slope and the second slope are not updated at this time, and the first slope and the second slope are continuously updated for the next data point; if the included angles of the two lines at this time are still greater than or equal to 180 degrees, the operation is stopped, and the data point whose included angles of the two lines are less than 180 degrees is taken as the termination point of the compression segment; if the included angles of the two lines at this time are less than 180 degrees, the pending data point is recorded as an outlier point, the first slope and the second slope are continuously updated for the next data point, and the termination point of the compression segment is obtained when the first slope and the second slope of two consecutive data points are both greater than or equal to 180 degrees, and the data points between the initial data point and the termination point are the first compression segment.

[0011] Preferably, the method for obtaining the threshold value of the next compression segment according to the threshold value and the outlier point of the previous compression segment is as follows: The last compression segment is recorded as a known compression segment, a line corresponding to the first slope of the known compression segment is recorded as an upper fulcrum line, a line corresponding to the second slope is recorded as a lower fulcrum line, the Euclidean distances of outliers of the known compression segment to the two fulcrum lines are calculated, the minimum Euclidean distance is recorded as an outlier distance of the outliers, a step length between the outliers of the known compression segment is obtained, the step length is the number of data points between the outliers plus one, and a threshold value of the next compression segment is obtained according to the outlier distance of the outliers, the step length between the outliers, the number of outliers and the threshold value of the known compression segment.

[0012] Preferably, the method for obtaining the threshold value of the next compression segment according to the outlier distance of the outliers, the step length between the outliers, the number of outliers and the threshold value of the known compression segment is: In the formula, the threshold value of the known compression segment is represented by, the outlier distance of the jth outlier in the cth compression segment is represented by, the number of outliers in the cth compression segment is represented by, the hyperparameter is represented by, the step length between the jth outlier and the next outlier in the cth compression segment is represented by, the number of data points in the cth compression segment is represented by, the exponential function with a natural constant as the base is represented by, the threshold value of the next compression segment of the known compression segment is represented by.

[0013] Preferably, the method for completing the compression storage of real-time data according to the threshold value of each compression segment and the outliers thereof is: the straight line data of each compression segment from the starting point to the ending point is taken as a first compression part, the outliers corresponding to each compression segment are taken as a second compression part, and the two compression parts are retained to complete the compression storage of real-time data.

[0014] The present application has the following beneficial effects: the present application proposes an information compression storage method for large-scale data analysis, outliers in data are obtained by analyzing the data, an initial compression segment is obtained by analyzing the whole real-time data, and the next compression segment is obtained based on the initial compression segment and the outliers thereof, the next compression segment is sequentially obtained based on the last compression segment and the outliers thereof, the threshold value is adaptively adjusted according to the change trend of the data, the compression performance of the rotating door compression algorithm is greatly improved without increasing the algorithm complexity. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0016] Figure 1 The flow chart of the information compression storage method for large-scale data analysis provided by an embodiment of the present application; Figure 2 The schematic diagram of an outlier; Figure 3 The schematic diagram of a traditional rotating door compression algorithm; Figure 4 The schematic diagram of the improved rotating door compression algorithm and its outlier. DETAILED DESCRIPTION

[0017] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purposes, the specific embodiments, structures, features and effects of the information compression storage method for large-scale data analysis according to the present application are described in detail as follows in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0019] Embodiment of the information compression storage method for large-scale data analysis: The specific scheme of the information compression storage method for large-scale data analysis provided by the present application will be specifically described below in combination with the drawings.

[0020] Please refer to Figure 1 which shows the flow chart of the information compression storage method for large-scale data analysis provided by an embodiment of the present application, and the method comprises the following steps: Step S001, acquiring real-time data segments and historical data sets.

[0021] The production equipment data is generally vibration data collected by sensors, current sensor data collected by current sensors, and voltage sensor data collected by voltage sensors. Each type of sensor data is recorded as one type of data. In this embodiment, one type of data is described. For each type of data, a data segment is collected every 10 minutes, and 10 minutes of data is collected each time. The collected data is recorded as a real-time data segment. Since the real-time data may be used for prediction and other aspects, the real-time data segment is transmitted to a real-time database. The real-time data segment is generally key data, and the data amount is small. The real-time data segment is not compressed in the real-time database. Data before the real-time data segment is stored in a historical database. When the real-time data segment is used up in the real-time database, the real-time database pushes a message to the historical database to transmit the real-time data segment to the historical database for long-term storage. At this time, the real-time data segment needs to be compressed. Each type of data is used as data support for historical alarm archiving and searching, historical curve viewing, intelligent alarm online or offline learning, and other functions. The data before compression in the historical database is recorded as historical data, and a set of all historical data is recorded as a historical data set.

[0022] At this time, the real-time data segment and the historical data set are obtained.

[0023] In step S002, the first data point of the real-time data segment is recorded as an initial data point. The abnormality degree of the initial data point is obtained according to the difference between the data value of the data point in the real-time data segment and the data value in the historical data set and the number of data points in the real-time data point. The initial threshold value is obtained according to the abnormality degree of the initial point.

[0024] The real-time data segment is compressed using a rotating door algorithm. The threshold value of the rotating door compression determines the compression ratio and the compression accuracy. The greater the threshold value, the more data points are included, which means that more data points are discarded, the compression ratio increases, but the compression error also increases synchronously. When the threshold value decreases, fewer data points are included, which means that fewer data points are discarded, the compression ratio decreases, but the compression error also decreases synchronously. The threshold value of the traditional rotating door algorithm is artificially given, and once the threshold value is determined, it cannot be changed in the entire compression process, and the controllability is poor. The particularity of data at different times is not considered. Therefore, the initial threshold value suitable for improving the compression accuracy is obtained in this embodiment.

[0025] For the first data point in the real-time data segment, it is taken as the reference point, the upper and lower points of the reference point with a threshold value are taken as the fulcrum of the rotating door, the second data point is connected with the two fulcrums, and the slope is calculated; then the third data point is connected with the two fulcrums, and the slope is calculated; the upper fulcrum above the reference point is recorded as the upper fulcrum, and the lower fulcrum below the reference point is recorded as the lower fulcrum, then the upper fulcrum is compared with the slope of different data points, and the straight line with larger slope is saved, and the lower fulcrum is compared with the slope of different data points, and the straight line with smaller slope is saved; then each data point is connected with the fulcrum to calculate the slope, and the slope comparison is performed respectively to save the straight line, until the angle between the straight line of the upper fulcrum and the straight line of the lower fulcrum is greater than or equal to 180 degrees, the operation is stopped, the previous data point is stored, and a new segment of compression is started from the point.

[0026] In the embodiment, the collected data is real-time data, and the amount of collected data is usually not large. The first data point in the real-time data segment is recorded as the initial data point. For each type of data, the data form of the initial data point is different, so it is necessary to adaptively obtain an initial threshold value according to the abnormality degree of the initial data point in the real-time data segment. The maximum data value and the minimum data value in the historical data set form a data interval, recorded as The abnormality degree of the initial data point is obtained according to the maximum value, the minimum value in the historical data set, and the data value of each data point in the real-time data segment and the number of data points in the data interval, and the formula is as follows: In the formula, represents the minimum data value in the historical data set, represents the maximum data value in the historical data set, represents the data value of the i-th data point in the real-time data point, represents the number of data points in the real-time data segment, represents the number of data points in the real-time data segment, represents the exponential function with the natural constant as the base, represents the abnormality degree of the initial data point.

[0027] If the data fluctuation degree in the real-time data point is larger, it means that the change trend of the data is faster and more obvious, and the device is more likely to have an abnormal operation. At the same time, the more the amount of data deviating from the normal operation interval, the more likely the device is in an abnormal operation state, so the importance of the data in the real-time data segment is greater, and the data can reflect the abnormal situation of the device operation, so the loss rate of the data should be as small as possible to facilitate the abnormal analysis of the device through the data, and therefore the threshold value should be smaller to ensure the compression accuracy of the data. The initial threshold value is obtained as follows: In the formula, This represents the threshold value for human experience compression during historical compression. Indicates the degree of anomaly of the initial data points. This represents the initial threshold value for the first compressed segment.

[0028] At this point, the initial threshold value has been obtained.

[0029] Step S003: Perform the first compression on the real-time data segment based on the initial threshold value to obtain the first compressed segment and its outliers. Obtain the threshold value of the second compressed segment based on the first compressed segment and its outliers. Obtain the threshold value of the second compressed segment and its outliers based on the threshold value of the second compressed segment. Obtain the threshold value of the third compressed segment based on the second compressed segment and its outliers, and so on. Obtain the threshold value of the next compressed segment based on the threshold value and outliers of the previous compressed segment.

[0030] The initial threshold value is the threshold value corresponding to the initial data points. In subsequent processing, the distribution and changes of the data will differ from the initial time. If the same threshold value is used globally, the compression accuracy and compression performance will be poor, and there will also be outliers in the data. Outliers will affect the compression performance of the preceding and following segments. Therefore, the threshold value should be adjusted adaptively according to the changes in the data. In order to balance compression accuracy and compression performance, it is necessary to analyze the data, obtain outliers in the data, and adaptively adjust the threshold value according to the changing trend of the data to adjust the compression accuracy.

[0031] Given a fixed initial threshold, outliers affect compression performance, such as... Figure 2 As shown, the 6th data point in the horizontal axis sequence is the outlier. This is because the 7th and 8th data points after the 6th data point are also within the initial threshold value. If a traditional rotating door compression algorithm is used for compression at this point, such as... Figure 3 As shown, the compressed segment 1 after compression consists of arrive The straight line replaces Data points, compressed segment 2 consists of arrive The straight line replaces Data points, compressed segment 3 consists of arrive The straight line replaces Data points, compressed segment 4 consists of arrive The straight line replaces Data points, among which This represents the (o+1)th data point; in this case, only the first data point is retained after compression. The 5th data point The 6th data point The 9th data point The 15th data point .

[0032] exist Figure 2 In the above, the 6th and 9th outliers are key factors affecting compression performance. If these outliers are marked and recorded separately, such as... Figure 4 As shown, the compressed data becomes compressed segment 1. arrive The straight line replaces Data points, compressed segment 2 consists of arrive The straight line replaces The data point, that is, the first data point retained after compression. The 10th data point The 15th data point .

[0033] Therefore, outliers are identified, and the threshold value of the next compression segment is adaptively adjusted based on the outlier degree and density of the outliers, as follows: Based on the initial threshold value The upper and lower threshold values ​​of the distance from the reference point are Using the two points as fulcrums of a revolving door, two virtual doors are established. Following the traditional revolving door rules, the termination point of the current compression segment is found. This termination point is the data point preceding the data point whose angle between the lines connecting the two fulcrums is greater than 180 degrees. Figure 3 Zhongru As shown, after determining the termination point, the second data point after the termination point is also connected to the fulcrum, and the maximum slope of the upper fulcrum and the minimum slope of the lower fulcrum are obtained respectively. If the angle between the two connecting lines is less than 180 degrees, the data point is considered to be within the compression segment as well. The data point after the termination point is recorded as an outlier. Figure 3 middle This is considered an outlier. If the angle between the two connecting lines is greater than 180 degrees, meaning neither of the two data points after the termination point is within the compressed segment, then the termination point is considered the termination point of the compressed segment, thus obtaining the first compressed segment. Figure 4 As shown, the 10th data point is... It is the end point of the compressed segment, and the two data points that follow are not in the first compressed segment.

[0034] At this time This refers to the end point of the first compression segment and the starting point of the next compression segment. There are also outliers in the first compression segment. Outliers are highly likely to be abnormalities in the production equipment. The threshold value of the next compression segment is adaptively adjusted according to the degree and density of outliers.

[0035] For the previous compression segment, the final connection between its two pivots is denoted as the upper pivot connection and the lower pivot connection, respectively. The equations of these two pivot connections are obtained, and the distances from the outliers to these two pivot connections are calculated. The minimum distance from the outlier to these pivot connections is recorded as the outlier distance. Based on the outlier distances, the initial threshold value, and the step size of the x-coordinates between the outliers, the threshold value for the next compression segment is obtained, as shown in the following formula: In the formula, This represents the threshold value of the c-th compressed segment. This represents the outlier distance of the j-th outlier point in the c-th compressed segment. This represents the number of outliers in the c-th compressed segment. This represents a hyperparameter; in this embodiment, the empirical value is 0.1. This represents the step size between the j-th outlier in the c-th compressed segment and the next outlier. This represents the number of data points in the c-th compressed segment. This represents an exponential function with the natural constant as its base. This represents the threshold value of the (c+1)th compressed segment.

[0036] In the above calculation, outliers belong to the data points in the compressed segment, and the step size represents the number of data points between two data points plus 1, i.e., as shown below. Figure 4 As shown, the 6th and 9th data points are discrete points, so the step size between them is 9-6=3. If there is only one discrete point in the compression segment, the last term in the formula is 1. If there is no discrete point in the compression segment, no adjustment is made, and the threshold value of the previous compression segment is used.

[0037] In the previous compression segment, the larger the deviation of outliers from the threshold value and the denser the distribution of outliers, the higher the probability of production equipment malfunction. Therefore, the accuracy requirement for subsequent data is higher, and the corresponding threshold value needs to be reduced. Similarly, the smaller the deviation of marked outliers from the threshold value and the sparser the distribution of outliers in the previous compression segment, the lower the probability of production equipment malfunction. Therefore, the accuracy requirement for subsequent data is lower, and the corresponding threshold value needs to be increased.

[0038] At this point, the threshold value of each compressed segment before compression has been obtained.

[0039] Step S004: Compress and store real-time data according to the threshold value of each compression segment and its outliers.

[0040] The data is compressed and stored using the above rules. The compressed data is divided into two parts: one part is the straight line data from the start point to the end point in the corresponding compressed segment, and the other part is the marked outlier data in the corresponding compressed segment. By compressing and storing real-time data, the compressed data is used as data support for functions such as historical alarm archiving and retrieval, historical curve viewing, and intelligent alarm online or offline learning, thereby monitoring, warning, and adaptively controlling the operation of production equipment.

[0041] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0042] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for information compression storage for large scale data analysis, characterized in that, The method comprises the following steps: acquire real-time data segment and historical data set; record the first data point of the real-time data segment as an initial data point, acquire the abnormality degree of the initial data point according to the difference between the data value of the data point in the real-time data segment and the data value in the historical data set and the number of data points in the real-time data point, and acquire the initial threshold value according to the abnormality degree of the initial point; acquire the first compressed segment and its outlier according to the first compression of the real-time data segment according to the initial threshold value, acquire the threshold value of the second compressed segment according to the first compressed segment and its outlier, acquire the second compressed segment and its outlier according to the threshold value of the second compressed segment, acquire the threshold value of the third compressed segment according to the second compressed segment and its outlier, and so on, and acquire the threshold value of the next compressed segment according to the threshold value and the outlier of the previous compressed segment; complete the compressed storage of the real-time data according to the threshold value and the outlier of each compressed segment.

2. The information compression storage method for large scale data analysis of claim 1, wherein, The method for acquiring the real-time data segment and the historical data set is: collect data every 1s for each type of data, collect data for a preset time, record the collected data as a real-time data segment, the real-time data segment is collected to the current time, and record all data before the real-time data segment as a historical data set.

3. The information compression storage method for large scale data analysis of claim 1, wherein, The method for acquiring the abnormality degree of the initial data point according to the difference between the data value of the data point in the real-time data segment and the data value in the historical data set and the number of data points in the real-time data point is: construct a data interval with the data values of all data points in the historical data set, the data interval is an interval from the minimum data value to the maximum data value in the historical data set, extract data points with data values in the data interval in the real-time data segment and record them as interval data points, and acquire the abnormality degree of the initial data point according to the ratio of the interval data points to all data points in the real-time data segment and the difference between the data value of the data point in the real-time data segment and the maximum data value and the minimum data value in the historical data set.

4. The information compression storage method for large scale data analysis of claim 3, wherein, The method for acquiring the abnormality degree of the initial data point according to the ratio of the interval data points to all data points in the real-time data segment and the difference between the data value of the data point in the real-time data segment and the maximum data value and the minimum data value in the historical data set is: wherein, represents the minimum data value in the historical data set, represents the maximum data value in the historical data set, represents the data value of the i-th data point in the real-time data points, represents the number of data points in the real-time data segment, represents the number of interval data points in the real-time data segment, represents the exponential function with the natural constant as the base, represents the abnormality degree of the initial data point.

5. The information compression storage method for large scale data analysis of claim 1, wherein, The method for acquiring the initial threshold value according to the abnormality degree of the initial point is: take the product of the abnormality degree of the initial data point and a preset compression threshold value as the initial threshold value of the real-time compressed segment.

6. The information compression storage method for large scale data analysis of claim 1, wherein, The method for acquiring the first compressed segment and its outlier according to the first compression of the real-time data segment according to the initial threshold value is: The point obtained by increasing the ordinate of the initial data point of the real-time data segment by the initial threshold value is recorded as an upper support point, the point obtained by decreasing the ordinate of the initial data point of the real-time data segment by the initial threshold value is recorded as a lower support point, a line is obtained by connecting the upper support point and the next data point, the slope of the line is calculated and recorded as a first slope, a line is obtained by connecting the lower support point and the next data point, the slope of the line is calculated and recorded as a second slope, the two angles between the two lines are obtained as the included angles of the two lines, when the included angles of the two lines corresponding to the first slope and the second slope are less than 180 degrees, a line is obtained by connecting the upper support point and the next data point, the slope of the line is calculated, the newly calculated slope and the first slope are stored, the maximum slope is recorded as a new first slope, a line is obtained by connecting the lower support point and the next data point, the slope of the line is calculated, the newly calculated slope and the second slope are stored, the minimum slope is recorded as a new second slope, and a first compression segment and corresponding outlier points are obtained according to the continuous updating of the first slope and the second slope.

7. The information compression storage method for large scale data analysis of claim 6, wherein, The method for obtaining a first compression segment and corresponding outlier points according to the continuous updating of the first slope and the second slope is as follows: When the included angles of the two lines corresponding to the first slope and the second slope are less than 180 degrees, the first slope and the second slope are updated again through the next data point, until the included angles of the two lines corresponding to the first slope and the second slope are greater than or equal to 180 degrees, the data point at this time is marked as a pending data point, the first slope and the second slope are not updated at this time, and then the first slope and the second slope are continuously updated for the next data point, if the included angles of the two lines at this time are still greater than or equal to 180 degrees, the operation is stopped, and the data point with the included angles of the two lines less than 180 degrees is recorded as an end point of the compression segment; if the included angles of the two lines at this time are less than 180 degrees, the pending data point is recorded as an outlier point, the first slope and the second slope are continuously updated for the next data point, until the first slope and the second slope of two continuous data points are both greater than or equal to 180 degrees, an end point of the compression segment is obtained, and the data points between the initial data point and the end point are a first compression segment.

8. The information compression storage method for large scale data analysis of claim 7, wherein, The method for obtaining a threshold value of a next compression segment according to a threshold value of a previous compression segment and an outlier point is as follows: The previous compression segment is recorded as a known compression segment, a line corresponding to the first slope of the known compression segment is recorded as an upper support point line, a line corresponding to the second slope is recorded as a lower support point line, the Euclidean distances of the outlier points of the known compression segment to the two support point lines are calculated, the minimum Euclidean distance is recorded as an outlier distance of the outlier point, a step length between the outlier points of the known compression segment is obtained, the step length is the number of data points between the outlier points plus one, and a threshold value of a next compression segment is obtained according to the outlier distance of the outlier point, the step length between the outlier points, the number of the outlier points and the threshold value of the known compression segment.

9. The information compression storage method for large scale data analysis of claim 8, wherein, The method for obtaining a threshold value of a next compression segment according to an outlier distance of an outlier point, a step length between the outlier points, a number of the outlier points and a threshold value of a known compression segment is as follows: wherein denotes the threshold value of the known compression segment, denotes the outlier distance of the jth outlier in the cth compression segment, denotes the number of outliers in the cth compression segment, denotes a hyperparameter, denotes the step length between the jth outlier and the next outlier in the cth compression segment, denotes the number of data points in the cth compression segment, denotes the exponential function with base of the natural constant, denotes the threshold value of the next compression segment of the known compression segment.

10. The information compression storage method for large scale data analysis of claim 1, wherein, The method for completing compression storage of real-time data according to the threshold value of each compression segment and the outlier points thereof is as follows: The straight line data from the starting point to the ending point of each compression segment is taken as a first compression part, and the corresponding outlier of each compression segment is taken as a second compression part, and the two compression parts are reserved to complete the compression storage of real-time data.

Images