Data compression method, system and equipment based on monitoring management platform and medium

Abstract

The invention relates to the technical field of data compression, in particular to a data compression method, system and device based on a monitoring management platform and a medium, and the method comprises the steps: obtaining monitoring point location data, carrying out the type analysis of the monitoring point location data, and obtaining the type result information; under the condition that the type result information is switching value information, the switching state change condition of the monitoring point location data is analyzed, and state change information used for storage is obtained; and under the condition that the type result information is analog quantity information, data statistics is performed on the monitoring point location data to obtain data statistical information for storage, so that the data management process of the intelligent power and environment monitoring management platform is optimized, the data storage cost is reduced, and the data storage efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data compression, and in particular to a data compression method, system, device and medium based on a monitoring management platform. Background Art

[0002] The intelligent dynamic environment monitoring and management platform is widely used in data centers and industrial automation fields, maintaining system stability through real-time monitoring and data collection.

[0003] At present, these intelligent dynamic environment monitoring and management platforms usually use a timed data collection method to record data from all monitoring points and store the data in a database. However, this method has some problems, such as high-frequency data collection generates a large amount of data, resulting in a significant increase in storage costs, and the database is prone to fill up quickly, requiring frequent hardware expansion or data migration. In addition, since the data is not effectively organized, the efficiency of querying and analyzing historical data is low, which affects the cost and user experience of the intelligent dynamic environment monitoring and management platform. The above problems need to be solved. Summary of the invention

[0004] In order to optimize the data management process of the intelligent dynamic environment monitoring and management platform, reduce the cost of data storage, and improve the efficiency of data storage, the present application provides a data compression method, system, device and medium based on the monitoring and management platform, which adopts the following technical solutions:

[0005] In a first aspect, the present application provides a data compression method based on a monitoring management platform, comprising:

[0006] Acquire monitoring point data, perform type analysis on the monitoring point data, and obtain type result information;

[0007] When the type result information is switch quantity information, the switch state change of the monitoring point data is analyzed to obtain the state change information for storage;

[0008] When the type result information is analog quantity information, data statistics are performed on the monitoring point data to obtain data statistical information for storage.

[0009] Preferably, the specific steps of performing data statistics on the monitoring point data to obtain data statistical information for storage are:

[0010] Perform data statistics on the monitoring point data to obtain the average value, maximum value and minimum value of the monitoring point data.

[0011] Preferably, it also includes:

[0012] Obtain division unit information, and perform data statistics on monitoring point data according to the division unit information;

[0013] Data statistics are used to obtain the average, maximum and minimum values ​​divided by minutes;

[0014] Data statistics are used to obtain the average, maximum and minimum values ​​divided by hours;

[0015] The data statistics are divided into daily average, maximum and minimum values.

[0016] Preferably, it also includes:

[0017] When the average value, maximum value and minimum value of the monitoring point data are obtained and stored, several secondary maximum values ​​of the monitoring point data are obtained and stored;

[0018] When the average value, the maximum value and the minimum value of the monitoring point data are obtained by statistics and stored, a plurality of secondary minimum values ​​of the monitoring point data are obtained and stored.

[0019] Preferably, it also includes:

[0020] The storage time information is obtained, and whether the state change information or the data statistical information has been stored for a timeout is determined according to the storage time information. If the state change information or the data statistical information has been stored for a timeout, the timed state change information or the data statistical information is cleared.

[0021] Preferably, the specific steps of analyzing the switch state change of the monitoring point data to obtain the state change information for storage are:

[0022] The current switch state information and the last switch state information that is adjacent to the current switch state information are obtained, and the current switch state information and the last switch state information are compared. If the states are different, the current switch state information for storage is obtained.

[0023] Preferably, it also includes:

[0024] The status change information and data statistics information are split into several historical data sub-tables for storage.

[0025] In a second aspect, the present application provides a data compression system based on a monitoring management platform, comprising:

[0026] The data acquisition module is used to acquire monitoring point data, perform type analysis on the monitoring point data, and obtain type result information;

[0027] The switch state analysis module is used to analyze the switch state change of the monitoring point data when the type result information is switch quantity information, and obtain the state change information for storage;

[0028] The data statistics module is used to perform data statistics on the monitoring point data when the type result information is analog information, and obtain data statistical information for storage.

[0029] In a third aspect, the present application provides a data compression device based on a monitoring management platform, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the data compression method based on the monitoring management platform as described above.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the data compression method based on the monitoring management platform as described above when running.

[0031] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0032] The present application obtains monitoring point data for type analysis and divides the monitoring point data into two types: switch quantity and analog quantity. When the type result information is switch quantity information, the switch state change of the monitoring point data is judged, and the state change information for storage is obtained according to the analysis, thereby reducing the amount of data storage. When the type result information is analog quantity information, data statistics are performed on the monitoring point data to obtain important data in the monitoring point data, that is, data statistical information is stored, thereby reducing the amount of data storage, thereby optimizing the data management process of the intelligent dynamic environment monitoring management platform, reducing the cost of data storage, and improving the efficiency of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a data compression method based on a monitoring management platform described in an embodiment of the present application.

[0034] Figure 2 It is a structural diagram of a data compression system based on a monitoring management platform described in an embodiment of the present application.

[0035] Description of reference numerals:

[0036] 1. Data acquisition module; 2. Switch state analysis module; 3. Data statistics module. DETAILED DESCRIPTION

[0037] The following combination Figure 1-Figure 2 The present application is described in further detail. The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting.

[0038] Reference Figure 1The present application relates to a data compression method based on a monitoring management platform, which specifically includes:

[0039] Step S1: Acquire monitoring point data, perform type analysis on the monitoring point data, and obtain type result information;

[0040] Step S2: when the type result information is switch quantity information, analyzing the switch state change of the monitoring point data to obtain the state change information for storage;

[0041] Step S3: When the type result information is analog quantity information, data statistics are performed on the monitoring point data to obtain data statistical information for storage.

[0042] Specifically, the present application obtains the monitoring point data monitored and collected in real time by the intelligent dynamic environment monitoring and management platform, performs a type analysis before storing it in the database, and when the type result information is switch quantity information, judges the switch state change of the monitoring point data, and obtains the state change information for storage according to the analysis, thereby reducing the amount of data storage. When the type result information is analog quantity information, the monitoring point data is statistically analyzed to obtain important data in the monitoring point data, that is, store the data statistical information, and reduce the amount of data storage, thereby optimizing the data management process of the intelligent dynamic environment monitoring and management platform, reducing the cost of data storage, and improving the efficiency of data storage.

[0043] Monitoring point data is divided into two types: switch data and analog data. Switch data refers to digital data that can only take two values, usually used to indicate the status of a device, such as on / off, run / stop, etc. It is relatively simple in data processing and is usually stored in binary form. Analog data represents continuous values ​​that can vary within a certain range, such as temperature, pressure, humidity, etc. Since this data is not discrete, it usually has a higher resolution and can take any value. For example, the temperature record can be 23.5℃, 23.6℃ or other values.

[0044] The acquisition of monitoring point data uses data tagging and classification for type analysis. In the data collection stage, the system marks the data type through configuration or preset rules. For example, the device configuration file or label is used to classify the data into switch quantity or analog quantity. Alternatively, the data processing algorithm is used for type analysis, and a specific data processing algorithm is used to automatically identify the data type. For example, the switch quantity and analog quantity are distinguished by the change pattern of the data. The switch quantity is usually a discrete state change, while the analog quantity data presents a smooth continuous change. Thus, the type result information is obtained, which is the classified monitoring point data.

[0045] As one implementation method, the specific steps of analyzing the switch state change of the monitoring point data and obtaining the state change information for storage are:

[0046] The current switch state information and the last switch state information that is adjacent to the current switch state information are obtained, and the current switch state information and the last switch state information are compared. If the states are different, the current switch state information for storage is obtained.

[0047] Specifically, the embodiment of the present application extracts the data of the current switch state from the system, for example, the current state of the light is "on" or "off", and then extracts the data of the last switch state that is adjacent to the current state, that is, finds the most recent record from the database, and compares the current switch state with the state of the last record. If the two are different, it means that the switch state has changed. In the case of a state change, the current switch state information is recorded, and the record will be stored as state change data.

[0048] The embodiment of the present application effectively reduces the storage requirements by storing only the data of state changes rather than the complete data of each acquisition point. Switching data is usually discrete. By comparing the current state with the previous state, the system only needs to record the change of state, avoiding repeated storage of the same state information, which not only reduces the amount of data storage, but also improves the storage efficiency and data processing speed of the system.

[0049] As one implementation method, the specific steps of performing data statistics on the monitoring point data to obtain data statistical information for storage are:

[0050] Perform data statistics on the monitoring point data to obtain the average value, maximum value and minimum value of the monitoring point data.

[0051] Specifically, the embodiment of the present application collects analog data within a specified time period from a monitoring management platform, and the data includes continuously changing values ​​such as temperature and pressure. Then, the average value of the collected analog data is calculated, and all data values ​​are added up and divided by the number of data points to obtain the average level of the data. Then, the maximum and minimum values ​​are calculated, and the maximum and minimum values ​​are identified in all the collected data. These two values ​​represent the boundaries of the data range and help understand the fluctuation range of the data. Finally, based on the above calculation results, data statistical information for storage is generated, including the average value, maximum value and minimum value. The statistical information is used to replace the original detailed data, thereby reducing storage requirements.

[0052] The embodiment of the present application also includes, after obtaining the data statistical information that replaces the original detailed data, cleaning the original detailed data, cleaning the analog original data that has been summarized and counted, and releasing storage space, thereby clearly recording the data status of the monitoring management platform while reducing the resources consumed by storage, which can significantly reduce storage costs, while retaining important data information and trends, and using the stored statistical data to provide a reference for subsequent data analysis and decision-making.

[0053] As one implementation mode, it also includes:

[0054] When the average value, maximum value and minimum value of the monitoring point data are obtained and stored, several secondary maximum values ​​of the monitoring point data are obtained and stored;

[0055] When the average value, the maximum value and the minimum value of the monitoring point data are obtained by statistics and stored, a plurality of secondary minimum values ​​of the monitoring point data are obtained and stored.

[0056] Specifically, when processing the statistics and storage of analog data, in addition to storing basic statistical indicators such as average, maximum and minimum values, the embodiment of the present application also performs additional data instance storage for subsequent query and analysis. The additional storage includes storing several secondary maximum values ​​and secondary minimum values.

[0057] This application first performs standard data collection and statistics, and counts and stores the average, maximum, and minimum values. On the basis of storing basic statistical information, additional data storage is performed to enable a more in-depth analysis of the details of the data.

[0058] The maximum value data is stored as follows: sort all monitoring point data, select the top 5 maximum values, and the other 4 data except the maximum value are secondary maximum values. Record the data instances of the 5 maximum values ​​for subsequent queries. The instances will provide extreme cases in the data to help understand the performance of the system under high load conditions and improve the user experience and efficiency of subsequent use.

[0059] The minimum value data is stored as follows: sort all monitoring point data, select the first 5 minimum values, and the other 4 data except the minimum value are secondary minimum values. Record the data instances of the 5 minimum values. The instance helps to analyze the performance of the system under low load conditions.

[0060] As one implementation mode, it also includes:

[0061] Obtain division unit information, and perform data statistics on monitoring point data according to the division unit information;

[0062] Data statistics are used to obtain the average, maximum and minimum values ​​divided by minutes;

[0063] Data statistics are used to obtain the average, maximum and minimum values ​​divided by hours;

[0064] The data statistics are divided into daily average, maximum and minimum values.

[0065] Specifically, in the process of statistically analyzing the monitoring point data, the embodiments of the present application process the data according to different time granularities, that is, the monitoring point data needs to be divided once according to the division unit information, so that the stored integrated data can be clearly understood during subsequent calls. The main time granularities include minutes, hours, and days. The average value, maximum value, and minimum value are divided once according to each division unit.

[0066] The analog data is counted by minute, hour, and day, and only the average, maximum, and minimum values ​​are saved. The storage period of the analog data summary is a custom number of days. The time period of data statistics refers to the time range for storing summary data. Users can define the period, for example, it can be set to the past 30 days, 60 days, or longer. This allows for sufficient statistical information when historical data needs to be reviewed.

[0067] The data statistics process is to first obtain raw data regularly from analog sensors or data sources, and deal with problems such as outliers and missing values ​​to ensure the accuracy of statistical results. Then, statistics are performed based on the division unit information. Statistics by minute: summarize the data for each minute and calculate the average, maximum, and minimum values ​​within the minute; statistics by hour: summarize the data for each hour and calculate the average, maximum, and minimum values ​​within the hour. Statistics by day: summarize the data for each day and calculate the average, maximum, and minimum values ​​within the day.

[0068] During the storage process, the data counted by minute, hour, and day are stored in the database. The data of each time granularity is stored separately to facilitate quick query and analysis.

[0069] As one implementation method, storage time information is obtained, and whether the state change information or data statistical information has been stored for a timeout is determined based on the storage time information. If the state change information or data statistical information has been stored for a timeout, the timed state change information or data statistical information is cleared.

[0070] Specifically, in the embodiment of the present application, the number of days is customized, and after the statistical data is saved for a certain period of time, the expired data can be automatically archived or deleted to optimize the storage space.

[0071] As one implementation mode, it also includes:

[0072] The status change information and data statistics information are split into several historical data sub-tables for storage.

[0073] Specifically, the embodiment of the present application adopts horizontal table partitioning technology to store state change information and data statistics information. Compared with the query process in the unpartitioned table state, the amount of data in the unpartitioned table is large, which may lead to low index efficiency and decreased query performance. The horizontal table partitioning technology is adopted to store historical data in tables to improve query efficiency. After the table is partitioned, the amount of data in a single table is reduced, the number of data rows that need to be scanned during query is reduced, and memory cache is often used. Table partitioning can cache more data pages and reduce disk I / O operations, thereby making data storage more reasonable and speeding up query speed.

[0074] Reference Figure 2 , an embodiment of the present application provides a data compression system based on a monitoring management platform, the system comprising:

[0075] The data acquisition module 1 is used to acquire monitoring point data, perform type analysis on the monitoring point data, and obtain type result information;

[0076] The switch state analysis module 2 is used to analyze the switch state change of the monitoring point data when the type result information is switch quantity information, and obtain the state change information for storage;

[0077] The data statistics module 3 is used to perform data statistics on the monitoring point data when the type result information is analog information, so as to obtain data statistics information for storage.

[0078] As one implementation method, the data statistics module performs data statistics on the monitoring point data, and the specific steps of obtaining the data statistics information for storage are:

[0079] Perform data statistics on the monitoring point data to obtain the average value, maximum value and minimum value of the monitoring point data.

[0080] As one implementation mode, it also includes:

[0081] The data statistics module is used to obtain the division unit information and perform data statistics on the monitoring point data according to the division unit information;

[0082] Data statistics are used to obtain the average, maximum and minimum values ​​divided by minutes;

[0083] Data statistics are used to obtain the average, maximum and minimum values ​​divided by hours;

[0084] The data statistics are divided into daily average, maximum and minimum values.

[0085] As one implementation mode, it also includes:

[0086] A data statistics module is used to obtain and store several secondary maximum values ​​of the monitoring point data after obtaining the average value, maximum value and minimum value of the monitoring point data through statistics.

[0087] When the average value, the maximum value and the minimum value of the monitoring point data are obtained by statistics and stored, a plurality of secondary minimum values ​​of the monitoring point data are obtained and stored.

[0088] As one implementation mode, it also includes:

[0089] The storage management module obtains storage time information, determines whether the state change information or data statistical information has been stored for a timeout according to the storage time information, and clears the timed state change information or data statistical information if the state change information or data statistical information has been stored for a timeout.

[0090] As one implementation mode, the switch state analysis module is used to analyze the switch state change of the monitoring point data, and the specific steps of obtaining the state change information for storage are:

[0091] The current switch state information and the last switch state information that is adjacent to the current switch state information are obtained, and the current switch state information and the last switch state information are compared. If the states are different, the current switch state information for storage is obtained.

[0092] As one implementation mode, it also includes:

[0093] The sub-table module is used to split the status change information and data statistical information into several historical data sub-tables for storage.

[0094] An embodiment of the present application provides a data compression device based on a monitoring management platform, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the data compression method based on the monitoring management platform as described above.

[0095] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the data compression method based on the monitoring management platform as described above when running.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0097] In several embodiments provided in this application, it should be understood that the disclosed methods, systems, devices and program products may be implemented in other ways.

[0098] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0099] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data compression method based on a monitoring management platform, characterized in that: include: Acquire monitoring point data, perform type analysis on the monitoring point data, and obtain type result information; When the type result information is switch quantity information, the switch state change of the monitoring point data is analyzed to obtain the state change information for storage; When the type result information is analog quantity information, data statistics are performed on the monitoring point data to obtain data statistical information for storage.

2. The data compression method based on the monitoring management platform according to claim 1 is characterized in that: The specific steps of performing data statistics on the monitoring point data to obtain data statistical information for storage are: Perform data statistics on the monitoring point data to obtain the average value, maximum value and minimum value of the monitoring point data.

3. The data compression method based on the monitoring management platform according to claim 2 is characterized in that: Also includes: Obtain division unit information, and perform data statistics on monitoring point data according to the division unit information; Data statistics are used to obtain the average, maximum and minimum values ​​divided by minutes; Data statistics are used to obtain the average, maximum and minimum values ​​divided by hours; The data statistics are divided into daily average, maximum and minimum values.

4. The data compression method based on the monitoring management platform according to claim 2 is characterized in that: Also includes: When the average value, maximum value and minimum value of the monitoring point data are obtained and stored, several secondary maximum values ​​of the monitoring point data are obtained and stored; When the average value, the maximum value and the minimum value of the monitoring point data are obtained by statistics and stored, a plurality of secondary minimum values ​​of the monitoring point data are obtained and stored.

5. The data compression method based on the monitoring management platform according to claim 1 is characterized in that: Also includes: The storage time information is obtained, and whether the state change information or the data statistical information has been stored for a timeout is determined according to the storage time information. If the state change information or the data statistical information has been stored for a timeout, the timed state change information or the data statistical information is cleared.

6. The data compression method based on the monitoring management platform according to claim 1 is characterized in that: The specific steps of analyzing the switch state change of the monitoring point data to obtain the state change information for storage are: The current switch state information and the last switch state information that is adjacent to the current switch state information are obtained, and the current switch state information and the last switch state information are compared. If the states are different, the current switch state information for storage is obtained.

7. The data compression method based on the monitoring management platform according to claim 1 is characterized in that: Also includes: The status change information and data statistics information are split into several historical data sub-tables for storage.

8. A data compression system based on a monitoring management platform, characterized in that: include: The data acquisition module is used to acquire monitoring point data, perform type analysis on the monitoring point data, and obtain type result information; The switch state analysis module is used to analyze the switch state change of the monitoring point data when the type result information is switch quantity information, and obtain the state change information for storage; The data statistics module is used to perform data statistics on the monitoring point data when the type result information is analog information, and obtain data statistical information for storage.

9. A data compression device based on a monitoring management platform, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the data compression method based on the monitoring management platform according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data compression method based on the monitoring management platform described in any one of claims 1 to 7 when running.

Images