Numerical monitoring data compression method, decompression method and monitoring platform

By setting the coordinates of the reference value and recording the deviation amount, compressed data packets are generated, which solves the problem of large consumption of numerical monitoring data storage space and transmission bandwidth, and realizes an efficient and secure data compression and decompression process.

CN118984157AActive Publication Date: 2024-11-19YILIAN CLOUD COMPUTING (HANGZHOU) CO LTD +1
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
CN202411471546.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-21
Publication Date
2024-11-19
Estimated Expiration
2044-10-21

AI Technical Summary

Technical Problem

The prior art is difficult to effectively compress and transmit a large amount of numerical monitoring data, resulting in large storage space consumption, high transmission bandwidth consumption and low response speed.

Method used

By receiving numerical monitoring data, building an array, setting a reference value, calculating the deviation between the element and the reference value, recording the coordinates of the non-zero deviation amount, and generating compressed data packets, including the coordinates of the reference value, non-zero deviation amount and deviation amount. , realize data compression and provide decompression methods to restore compressed data.

Benefits of technology

A higher data compression ratio is achieved, data storage space is reduced, data transmission efficiency and security are improved, and the losslessness of the compression and decompression process is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984157A_ABST
    Figure CN118984157A_ABST
Patent Text Reader

Abstract

The invention discloses a compression method and decompression method of numerical type monitoring data and a monitoring platform, and the compression method of the numerical type monitoring data comprises the steps: receiving the numerical type monitoring data, and constructing an array; judging whether the data type of the data in the array is a telemetering data type or a remote pulse data type, setting corresponding reference values according to different data types of the elements in the array, calculating deviations between the elements in the array and the reference values, recording coordinates of the elements with non-zero deviations, and storing the coordinates of the elements in the array; the reference value is configured to enable the proportion of the elements with the non-zero deviation values in the total elements to be smaller than a preset proportion; a compressed data packet is generated, the compressed data packet including a reference value, a non-zero deviation amount, and coordinates of an element for which the deviation amount is non-zero. By adopting the method and the device, a higher data compression ratio can be realized, the data storage space can be reduced, the losslessness of the compression process can be ensured, the data does not lose any information in the compression process, and the efficiency and the safety of data compression are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a compression method, a decompression method and a monitoring platform for numerical monitoring data. Background Art

[0002] With the advancement of technology and the deepening of industrial applications, the amount of measurement point data that needs to be processed is increasing. This is especially true in industries such as electricity, energy storage, water, and gas, where data from tens of thousands or even more measurement points often needs to be collected, processed, transmitted, and stored. However, large amounts of raw data not only take up a large amount of storage space, but also consume a lot of bandwidth and time during transmission. Furthermore, the response speed is slow when parsing the databases, applications, cloud service platforms, and other objects that store this raw data. Therefore, research on efficient data compression and transmission technologies has important practical significance and application value. Summary of the Invention

[0003] In order to address the deficiencies of the prior art, the purpose of this application is to provide a compression method, a decompression method and a monitoring platform for numerical monitoring data, which can achieve a higher data compression ratio and reduce data storage space.

[0004] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application provides a method for compressing numerical monitoring data, the method comprising the following steps: receiving the numerical monitoring data, and constructing an array, using the numerical monitoring data as an element of the array; Determining whether the data type of the data in the array is a telemetry data type or a telepulse data type, and setting corresponding reference values ​​according to the data types of the elements in the array, wherein, in response to determining that the data type is a telemetry data type, setting the median of each tuple in the array as the reference value of the tuple data; and in response to determining that the data type is a telepulse data type, setting the sum of the first element of each tuple in the array and n-1 times the average growth amount of the data in the tuple as the reference value of the nth element in the tuple; Calculating the deviation between each element in the array and the reference value, and recording the coordinates of the elements with non-zero deviations, wherein the reference value is configured so that the proportion of the elements with non-zero deviations in the total elements is less than a preset proportion; A compressed data packet is generated, the compressed data packet including the reference value, the non-zero offset, and the coordinates of the elements whose offsets are non-zero.

[0005] The above-mentioned compression method of numerical monitoring data receives numerical monitoring data and records the numerical monitoring data in the form of an array; and sets different reference values ​​for different data types of telemetry and telepulse, calculates the deviation between each element in the array and the reference value, and records the coordinates of the elements with non-zero deviation. The reference value is configured to make the proportion of elements with non-zero deviation in the total elements less than a preset proportion, thereby reducing the data volume of non-zero elements; and then generates a compressed data packet, which includes the reference value, non-zero deviation, and the coordinates of the elements with non-zero deviation, thereby achieving a higher data compression ratio, reducing data storage space, and ensuring the losslessness of the compression process, so that no information is lost during the compression process, thereby improving the efficiency and security of data compression.

[0006] In one embodiment, the method further includes: recording a first moment reference value, calculating a first deviation between each data in a tuple at a second moment and the first moment reference value, and counting the number of elements whose first deviation is non-zero to obtain a first number of non-zero elements; calculating a second moment reference value, calculating a second deviation between each data in a tuple at a second moment and the second moment reference value, and counting the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; comparing the first number of non-zero elements and the second number of non-zero elements, and in response to the first number of non-zero elements being greater than or equal to the second number of non-zero elements, taking the second moment reference value as the reference value of the tuple at the second moment, and in response to the first number of non-zero elements being less than the second number of non-zero elements, taking the first moment reference value as the reference value of the tuple at the second moment.

[0007] In one embodiment, the method further includes: recording a reference value at a first moment, calculating a first deviation between each data in the tuple at a second moment and the reference value at the first moment, and counting the number of elements for which the first deviation is non-zero to obtain a first number of non-zero elements; Calculating a second-time reference value, calculating a second deviation between each data in the tuple at the second time and the second-time reference value, and counting the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; Calculating a benchmark mean, where the benchmark mean is a weighted sum of the benchmark value at the first moment and the benchmark value at the second moment, calculating a third deviation between each data in the tuple at the second moment and the benchmark mean, and counting the number of elements for which the third deviation is non-zero to obtain a third number of non-zero elements; Compare the first number of non-zero elements, the second number of non-zero elements, and the third number of non-zero elements, and take the first moment reference value, the second moment reference value, and the reference mean value that minimizes the number of non-zero elements as the reference value of the tuple at the second moment.

[0008] In one embodiment, in response to the numerical monitoring data fluctuating within a preset range, the data type is determined to be a telemetry data type, and in response to the numerical monitoring data accumulating in a time series, the data type is determined to be a telepulse data type.

[0009] In the second aspect, the present application provides a decompression method for numerical monitoring data, which is used to restore a compressed data packet generated by a compression method for numerical monitoring data as described in any one of the first aspects. The decompression method includes the following steps: generating a restored data array; obtaining the coordinates of elements in the compressed packet whose deviation is non-zero, filling the restored data in the corresponding positions of the restored data array, the restored data being the sum of the reference value and the deviation, and filling the reference value in the remaining positions of the restored data array.

[0010] The above-mentioned decompression method of numerical monitoring data generates a restored data array from the data compression package generated by the above-mentioned compression method of numerical monitoring data, obtains the coordinates of the elements with non-zero deviation in the compressed package, fills the restored data in the corresponding positions of the restored data array, and the restored data is the sum of the reference value and the deviation. The reference value is filled in the remaining positions of the restored data array, ensuring the losslessness of the decompression process, so that no information is lost during the decompression process, and improving the efficiency and security of data decompression.

[0011] In a third aspect, the present application further provides a monitoring platform, the monitoring platform comprising: a collection module, the collection module being at least used to collect numerical monitoring data; A processing module, the processing module is used to construct an array, and the numerical monitoring data is used as an element of the array; The processing module is further configured to determine whether the data type of the elements in the array is a telemetry data type or a telepulse data type, and to set corresponding reference values ​​according to the different data types of the elements in the array, wherein, in response to determining that the data type is a telemetry data type, the median of each tuple in the array is set as the reference value of the tuple data; and in response to determining that the data type is a telepulse data type, the sum of the first element of each tuple in the array and n-1 times the average growth amount of the data in the tuple is set as the reference value of the nth element in the tuple. The processing module is further configured to calculate the deviation between each element in the array and the reference value, and record the coordinates of the elements with non-zero deviations, wherein the reference value is configured so that the proportion of the elements with non-zero deviations in the total elements is less than a preset proportion; The processing module is configured to generate a compressed data packet, wherein the compressed data packet includes the reference value, the non-zero offset, and the coordinates of the elements whose offsets are non-zero.

[0012] The above-mentioned monitoring platform collects numerical monitoring data through the acquisition module, and records the numerical monitoring data in the form of an array through the processing module; the processing module is also used to set a reference value, calculate the deviation between each element in the array and the reference value, and record the coordinates of the elements with non-zero deviation. The reference value is configured to make the proportion of elements with non-zero deviation in the total elements less than a preset proportion; the processing module is configured to generate a compressed data packet, which includes the reference value, the non-zero deviation, and the coordinates of the elements with non-zero deviation, thereby achieving a higher data compression ratio, ensuring the losslessness of the compression process, and preventing any information from being lost during the compression process, thereby improving the efficiency and security of data compression.

[0013] In one embodiment, the processing module is also used to record the first moment reference value, calculate the first deviation between each data in the tuple at the second moment and the first moment reference value, and count the number of elements whose first deviation is non-zero to obtain the first number of non-zero elements; the processing module calculates the second moment reference value, calculates the second deviation between each data in the tuple at the second moment and the second moment reference value, and counts the number of elements whose second deviation is non-zero to obtain the second number of non-zero elements; the processing module compares the first number of non-zero elements and the second number of non-zero elements, and in response to the first number of non-zero elements being greater than or equal to the second number of non-zero elements, takes the second moment reference value as the reference value of the tuple at the second moment, and in response to the first number of non-zero elements being less than the second number of non-zero elements, takes the first moment reference value as the reference value of the tuple at the second moment.

[0014] In one embodiment, the processing module is further configured to record a reference value at a first moment, calculate a first deviation between each data in the tuple at a second moment and the reference value at the first moment, and count the number of elements for which the first deviation is non-zero to obtain a first number of non-zero elements; The processing module calculates a second-time reference value, calculates a second deviation between each data in the tuple at the second time and the second-time reference value, and counts the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; The processing module calculates a benchmark mean, where the benchmark mean is obtained by taking a weighted sum of a benchmark value at the first moment and a benchmark value at the second moment, calculates a third deviation between each data in the tuple at the second moment and the benchmark mean, and counts the number of elements whose third deviation is non-zero to obtain a third number of non-zero elements; The processing module compares the sizes of the first number of non-zero elements, the second number of non-zero elements and the third number of non-zero elements, and takes the first moment reference value, the second moment reference value and the reference mean that minimizes the number of non-zero elements as the reference value of the tuple at the second moment.

[0015] In one embodiment, the processing module uses the decompression method of the numerical monitoring data as described in the second aspect to restore the compressed data packet.

[0016] In one embodiment, the monitoring platform includes a storage module, which includes a hot data cache area and a cold data cache area. In the cold data cache area, the numerical monitoring data is stored in the form of the compressed data packet. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an application scenario diagram of a method for compressing numerical monitoring data in an embodiment of the present application; Figure 2 Schematic diagram of a flow chart of a method for compressing numerical monitoring data in an embodiment of the present application; Figure 3 Schematic diagram of a flow chart of another method for compressing numerical monitoring data in an embodiment of the present application; Figure 4 This is a flowchart of a method for decompressing numerical monitoring data according to one embodiment of the present application; Figure 5 This is a schematic diagram of the structure of a monitoring platform in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the specific implementation of the present application will be clearly and completely described below in conjunction with the drawings in the implementation of the present application.

[0019] With the advancement of technology and the deepening of industrial applications, the amount of measurement data that needs to be processed is increasing. This is especially true in large-scale engineering projects, environmental monitoring, smart grids, and other fields, where data from tens of thousands or even more measurement points often needs to be collected, processed, and transmitted in real time. However, this large amount of raw data not only takes up a significant amount of storage space, but also consumes a significant amount of bandwidth and time during transmission. Therefore, research on efficient data compression and transmission technologies has important practical significance and application value.

[0020] In order to improve the compression efficiency and data transmission efficiency of data, the present application provides a compression method and a decompression method for numerical monitoring data. The compression method can receive numerical monitoring data and record the numerical monitoring data in the form of an array; set a reference value, calculate the deviation between each element in the array and the reference value, record the coordinates of the elements with non-zero deviation, and the reference value is configured to make the proportion of the elements with non-zero deviation in the total elements less than a preset proportion; and then generate a compressed data packet including the non-zero deviation, the coordinates of the elements with non-zero deviation, and the reference value to achieve data compression. The decompression method can generate a restored data array, obtain the coordinates of the elements with non-zero deviation in the compressed packet, fill the restored data in the corresponding position of the restored data array, the restored data is the sum of the reference value and the deviation, and fill the reference value in the remaining positions of the restored data array to achieve data decompression. Thus, the compression method and decompression method of the numerical monitoring data of the present application can achieve the reduction of data volume to reduce data storage space without losing useful information, thereby reducing data transmission bandwidth and improving data transmission efficiency.

[0021] The compression method and decompression method of the numerical monitoring data of the present application can be applied to Figure 1 In the application scenario shown. The application scenario includes a smart device 101, a data acquisition device 102, a database 103, a data compression device 104, and a data decompression device 105. Among them, the smart device 101 includes but is not limited to energy storage devices, industrial and commercial energy storage cabinets, smart gas meters, etc. The data acquisition device 102 includes but is not limited to sensors or converters. The data acquisition device 102 can perform real-time monitoring and data acquisition on the smart device 101, and store the collected data in the database 103. The data compression device 104 can use the compression method for numerical data provided in this application to compress the data, generate a data compression package, and transmit the data to the data decompression device 105 to improve the data compression ratio, reduce the data transmission bandwidth, and improve the data transmission efficiency. The data decompression device 105 can use the decompression method for numerical data provided in this application to decompress the compressed package to ensure the losslessness of the data, improve the data decompression efficiency and the security of the data.

[0022] It should be noted that Figure 1 This is only an example of an application scenario of this application. The above description is intended to more clearly illustrate the technical solution of the embodiment of this application and does not constitute a limitation of the technical solution provided by the embodiment of this application. Those skilled in the art will appreciate that with the evolution of various devices and the emergence of new application scenarios, the technical solution of the embodiment of this application is also applicable to similar technical problems.

[0023] The following describes in detail the compression and decompression methods of numerical monitoring data in the embodiments of the present application.

[0024] like Figure 2 As shown, the compression method of numerical monitoring data provided in the embodiment of the present application specifically includes the following steps: Step S201: Receive numerical monitoring data and construct an array, using the numerical monitoring data as elements of the array.

[0025] Among them, numerical monitoring data represents the observed values ​​measured according to the digital scale. Numerical monitoring data can be divided into numeric type, floating point type, double precision type and integer type according to storage, representation form and value range.

[0026] Specifically, an array can be considered as naming a finite set of variables of the same type. It is an ordered sequence of elements. An array can include multiple tuples, and each tuple contains multiple ordered elements. The array is recorded in the form of Contains m Line and n Each row or column represents a tuple. Numerical monitoring data is recorded in the form of an array to facilitate subsequent classification and processing of various types of data collected at that moment.

[0027] For example, a certain numerical monitoring data is shown in Table 1 below. The numerical monitoring data includes 4 arrays, and each array includes 3 elements.

[0028] Table 1

[0029] Then the numerical monitoring data can be represented as an array : Expression (1).

[0030] Step S202: set a reference value, calculate the deviation between each element in the array and the reference value, record the coordinates of the elements with non-zero deviation, and configure the reference value so that the proportion of elements with non-zero deviation in the total elements is less than a preset proportion.

[0031] The reference value can be set based on empirical values ​​or the median, mode, or other factors of each tuple in the array. The reference value is set so that the proportion of elements in the array with a non-zero deviation from the reference value is less than a preset proportion of the total elements. Preferably, the reference value is configured so that the number of elements with a non-zero deviation is less than the number of elements with a zero deviation, thereby further improving the data compression ratio (also known as the compression factor, i.e., original data size divided by compressed data size).

[0032] For example, based on the array of numerical monitoring data in Table 1 , get the mode of each tuple as 80, 8, 220, 35, calculate the array The deviation between each element and the reference value, each deviation is represented as an array : Expression (2).

[0033] Observation Array It can be seen that there are 4 non-zero deviations and 12 total elements, that is, the proportion of non-zero deviations in the total elements is 4 / 12.

[0034] Furthermore, the elements with non-zero offsets and the coordinates of non-zero elements can be recorded in the form of a triple [values, row_indices, col_indices], where values ​​represents the non-zero offset, row_indices represents the row index corresponding to the non-zero element, and col_indices represents the column index corresponding to the non-zero element. Therefore, the coordinate (0, 0) represents the element in the first row and first column of the array, then the array The non-zero elements in each deviation and the coordinates of the non-zero elements are expressed in the form of triplets as follows: [6, 0, 2], [1, 1, 2], [-10, 2, 0], [2, 3, 2].

[0035] Step S203: Generate a compressed data packet, which includes a reference value, a non-zero offset, and the coordinates of elements with non-zero offsets.

[0036] Among them, the data compression package generated based on the reference value, the non-zero deviation, and the coordinates of the element with the non-zero deviation is a data package generated by compressing the data by reducing the data volume and storage space without losing useful information.

[0037] The generated data compression package can be expressed as: {criteria=baseline value, count=number of elements, values=[v1, ..., v n ], row_indices=[r1,...,r n ], col_indices=[c1,...,c n ]}, values ​​contains all elements with non-zero deviations, row_indices represents the row index corresponding to the non-zero elements in values, and col_indices represents the column index corresponding to the non-zero elements in values.

[0038] For example, based on the above array and arrays The resulting compressed data packet can be represented as Tuple = {criteria = [80, 8, 220, 35], count = 12, values ​​= [6, 1, -10, 2], row_indices = [0, 1, 2, 3], col_indices = [2, 2, 0, 2]}. You can also see that while 12 elements were originally transmitted, after compression, only the four elements with non-zero deviations need to be expressed, significantly reducing the data size.

[0039] It should be noted that, in one embodiment, the above-mentioned data compression package can be represented in binary, that is, [values ​​(non-zero deviation), row_indices (row index corresponding to non-zero elements), col_indices (column index corresponding to non-zero elements)] is translated into binary and stored for subsequent reading and calling.

[0040] According to the above description, the compression method of numerical monitoring data of this embodiment sets a reference value according to the characteristics of the numerical monitoring data, and obtains the deviation between each element in the numerical monitoring data array and the reference value, so that the reference value is configured to make the proportion of elements with non-zero deviation in the total elements less than a preset proportion, so as to reduce the number of non-zero elements, and then generate a compressed data packet through the reference value, the non-zero deviation, and the coordinates of the elements with non-zero deviation, thereby reducing the amount of data to reduce data storage space without losing useful information, achieving a higher data compression ratio, improving the efficiency and security of data compression, and further reducing the data transmission bandwidth and improving data transmission efficiency.

[0041] In one embodiment, the method for compressing numerical monitoring data further includes: determining whether the data type of the data in the array is a telemetry data type, and in response to the data type being a telemetry data type, setting the median of each tuple in the array as a baseline value of the tuple data.

[0042] The telemetry data type represents real-time remote monitoring data at a specific moment. Telemetry refers to the monitoring and transmission of device parameters through remote data transmission. For example, in the energy storage industry, telemetry can be used to monitor device parameters such as voltage, current, temperature, remaining charge, and battery health, which fluctuate within a preset range. The median represents the middle number in a sequentially arranged set of data.

[0043] For example, take the telemetry data of the battery cell group of the industrial and commercial energy storage cabinet as an example. The industrial and commercial energy storage cabinet collects and reports data from several telemetry collection points with the same attributes. Taking the data of 400 battery cells in this scenario as an example, each battery cell reports four groups of numerical monitoring data: remaining power, battery health, voltage, and temperature. Therefore, the numerical monitoring data can be represented as an array : Expression (3); In the formula, [ ,…, ] represents 400 data in a tuple, where i is 1, 2, 3, and 4.

[0044] Therefore, the array The data type of the data in is the telemetry data type, assuming that the tuple [ ,…, The median of ] is , then the base value of the tuple is . Similarly, you can get an array The base values ​​of each tuple in are 、 、 、 , and then calculate the deviation between each element in the array and the reference value, and get the array of deviations . Since the design of the industrial and commercial energy storage cabinet requires that the battery cells have consistency, and the management system of the industrial and commercial energy storage cabinet will balance the battery cells to keep the parameter information of each battery cell consistent to ensure the stable operation of the industrial and commercial energy storage cabinet, the parameter information of the 400 battery cells are relatively similar. When the median is used to make a difference with each data, most of the deviations obtained are zero, and the compression method of this embodiment only needs to compress non-zero elements, thereby reducing the amount of data that needs to be compressed. According to experimental data, the maximum compression ratio in this scenario can reach 200 times, and the average compression ratio is about 20 times, while the compression ratio of conventional compression methods (such as Huffman coding and LZW compression) is about 2 times. It can be seen that the compression method of this embodiment greatly improves the compression ratio of the data.

[0045] In this embodiment, when the data type of the data in the array is a telemetry data type, the median of each tuple in the array is set as the baseline value of the tuple data, and then based on the baseline value, the deviation between each element in the array and the baseline value can be calculated, so that the proportion of elements with non-zero deviation in the total elements is less than a preset proportion, thereby further improving the data compression multiple, reducing data storage space, and further saving transmission bandwidth, thereby improving data transmission efficiency and real-time performance.

[0046] In one embodiment, the compression method of numerical monitoring data also includes: determining whether the data type of the data in the array is a remote pulse data type, and in response to the data type being a remote pulse data type, setting the sum of the first element of each tuple in the array and the average growth amount of n-1 times the data in the tuple as the baseline value of the nth element in the tuple.

[0047] The remote pulse data type indicates remote monitoring data acquired at a specific moment and accumulated within a preset time range prior to that moment, expressed as data accumulated in a time series. Remote pulse refers to the transmission of device metering data via remotely transmitted pulse signals. For example, in the energy storage industry, remote pulse can be used to transmit metering information such as the charge and discharge capacity, accumulated energy, and other energy storage device information. In this scenario, the tuples in the array at a specific moment contain the charge and discharge capacity, or accumulated energy data, of the energy storage device within the preset time range prior to that moment.

[0048] When the data type is the remote pulse data type, the reference value of each data in each tuple in the array can be expressed as the following relationship: Expression (4); Where, Indicates the first The base value of the bit element, Represents the first element, represents the average growth of the data in the tuple, and, , Indicates the total amount of data in the tuple.

[0049] For example, taking the cumulative charge and discharge remote pulse data of the industrial and commercial energy storage cabinet as an example, 300 cumulative charge and discharge data with a 5-minute reporting interval and a second-level collection frequency are used as an example to illustrate the receiving of the cumulative charge and cumulative discharge group numerical monitoring data. Therefore, the numerical monitoring data can be represented as an array : Expression (5); Therefore, the array The data type of the data is the remote pulse data type, tuple [ ,…, The first element of ] is , assuming that the average growth of the data in the tuple is , then the benchmark values ​​of each data in the tuple are 、 + 、 + ×2… + ×199. Similarly, we can get the tuple [ ,…, The average growth of data in ] is , the benchmark values ​​of each data in the tuple are 、 + 、 + ×2… + ×199, and then calculate the deviation between each element in the array and the reference value, and you can get the array of deviations. .

[0050] If, in this scenario, the industrial and commercial energy storage cabinet does not perform any charge or discharge operations during this period, the average growth is 0, and the baseline value is the first element plus 0. Then, the difference between the baseline value and each element is calculated to obtain the array of deviations. It is empty, that is, there are no non-zero elements that need to be compressed. In this case, the most ideal compression state can be achieved.

[0051] If, in this scenario, the industrial and commercial energy storage cabinet continues to charge and discharge evenly during this period, the average growth is (last element - first element) / (total data volume - 1), and the baseline value is the first element plus the average growth. Then, the difference between the baseline value and each element is taken to obtain the array of deviations. There are only a few non-zero elements. In this case, the data compression ratio is greatly improved.

[0052] In this embodiment, according to the data characteristics of the remote pulse data, the sum of the first element of each tuple in the array and the average growth of n-1 times the data in the tuple is set as the baseline value of the nth element in the tuple, and then based on the baseline value, the deviation between each element in the array and the baseline value is calculated, so that the proportion of elements with non-zero deviation in the total elements is less than the preset proportion, thereby further improving the data compression ratio and reducing data storage space.

[0053] In order to improve the data compression ratio and reduce the data storage space, in one embodiment, Figure 3 As shown, the compression method of numerical monitoring data also includes: Step S301: record the first moment reference value, calculate the first deviation between each data in the tuple at the second moment and the first moment reference value, and count the number of elements whose first deviation is non-zero to obtain the first non-zero element number.

[0054] Among them, if the data type of the data in the array is a telemetry data type, each data in the tuple at a certain moment represents the numerical monitoring data received in real time at that moment, and the interval time between the first moment and the second moment can be set to a unit time of 1 second.

[0055] If the data type of the data in the array is the remote pulse data type, the tuple at a certain moment contains the historical data of the target object within a certain time range before the moment, and the interval between the first moment and the second time can be set to 5 minutes.

[0056] For example, assuming that the data types at the first moment and the second moment are telemetry data types, the data at the first moment is an array : Expression (6); The data at the second moment is an array : Expression (7); Then, the first moment reference values ​​can be obtained as 81 and 18, and the first deviation between each data in the tuple at the second moment and the first moment reference value can be calculated and expressed as an array : Expression (8).

[0057] Array of first deviation statistics The number of non-zero elements in is 4.

[0058] Step S302: Calculate the second moment reference value, calculate the second deviation between each data in the tuple at the second moment and the second moment reference value, and count the number of elements whose second deviation is non-zero to obtain the second number of non-zero elements.

[0059] For example, the array at the second moment is calculated The reference values ​​are 81 and 20, and the second deviation between each data in the tuple at the second moment and the reference value at the second moment is calculated and expressed as an array : Expression (9).

[0060] Then, the second deviation array is counted The number of non-zero elements in is obtained, and the second number of non-zero elements is 2.

[0061] Step S303: Compare the first number of non-zero elements and the second number of non-zero elements. In response to the first number of non-zero elements being greater than or equal to the second number of non-zero elements, take the second moment reference value as the reference value of the tuple at the second moment. In response to the first number of non-zero elements being less than the second number of non-zero elements, take the first moment reference value as the reference value of the tuple at the second moment.

[0062] Exemplarily, by comparing the first number of non-zero elements and the second number of non-zero elements, it can be seen that 4 is greater than 2, so the second moment reference value is taken as the reference value of the tuple at the second moment.

[0063] Similarly, if the data types at the first moment and the second moment are remote pulse data types, the processing process thereof refers to the processing process of the remote measurement data type, which will not be described in detail here.

[0064] In this embodiment, the first and second non-zero element counts are obtained by recording data and reference values ​​at the first and second moments. When the first non-zero element count is greater than or equal to the second non-zero element count, the reference value at the second moment is used as the reference value for the tuple at the second moment. When the first non-zero element count is less than the second non-zero element count, the reference value at the first moment is used as the reference value for the tuple at the second moment. This allows for obtaining optimal reference values ​​based on the characteristics of different data, maximizing data compression, improving the data compression ratio, and reducing data storage space.

[0065] In another embodiment, the method for compressing numerical monitoring data further includes: Calculating a benchmark mean, where the benchmark mean is obtained by taking a weighted sum of the benchmark value at the first moment and the benchmark value at the second moment, calculating a third deviation between each data in the tuple at the second moment and the benchmark mean, and counting the number of elements whose third deviation is non-zero to obtain a third number of non-zero elements; Compare the first number of non-zero elements, the second number of non-zero elements, and the third number of non-zero elements, and take the first moment reference value, the second moment reference value, and the reference mean value that minimizes the number of non-zero elements as the reference value of the tuple at the second moment.

[0066] In this embodiment, the first, second, and third numbers of non-zero elements are obtained by recording the baseline values ​​and baseline mean at the first and second moments. The baseline value at the first, second, and baseline mean values ​​that results in the least number of non-zero elements is then used as the baseline value for the tuple at the second moment. This allows for optimal baseline values ​​to be obtained based on the characteristics of different data, maximizing data compression, improving the data compression ratio, and reducing data storage space.

[0067] Based on the same inventive concept, the present application also provides a method for decompressing numerical monitoring data, which is used to restore a compressed data packet generated by the compression method for numerical monitoring data in any of the above embodiments, such as Figure 4 As shown, the decompression method specifically includes the following steps: Step S401: Generate a restored data array.

[0068] Step S402: Obtain the coordinates of the elements with non-zero deviation in the compressed package, fill the corresponding positions of the restored data array with restored data, the restored data being the sum of the reference value and the deviation, and fill the remaining positions of the restored data array with the reference value.

[0069] For example, based on the array tuple Tuple={criteria=[80, 8, 220, 35], count=12, values=[6, 1, -10, 2], row_indices=[0, 1, 2, 3], col_indices=[2, 2, 0, 2]} of the above data compression package, first determine the array of deviations, as well as the coordinates and reference values ​​of non-zero elements according to count, values, row_indices, and col_indices in the data compression package, then fill in the restored data according to the sum of the reference value and the deviation, and fill in the reference value in the remaining positions, so that the generated restored array is the array .

[0070] In this embodiment, a restored data array is generated by converting the data compression package generated by the above-mentioned compression method of numerical monitoring data, and the coordinates of the elements in the compressed package with non-zero deviation are obtained, and the restored data are filled in the corresponding positions of the restored data array. The restored data is the sum of the reference value and the deviation, and the reference value is filled in the remaining positions of the restored data array to ensure the losslessness of the decompression process, so that no information is lost during the decompression process, thereby improving the efficiency and security of data decompression.

[0071] In one embodiment, the decompression method further includes: storing the time series data of the data compression package in a time series database, defining the time cache range within 24 hours, and the data with more than 20 access times in the past 7 days as hot data, and defining the remaining data as cold data, storing the cold data in the time series database in the form of a compressed data package, and decompressing it when needed.

[0072] Specifically, hot data can be cached in memory or fast storage media, ensuring low query latency and efficient query performance. Cold data is stored in compressed form and decompressed when needed, reducing storage requirements and improving data security. When querying or analyzing data, historical time series data for the corresponding telemetry or telepulse data can be pulled, and the compressed data packages can be cyclically decompressed based on the time series data for display and analysis. Paging queries can be used during querying, and compressed data packages can be decompressed in memory, saving storage space and improving data decompression efficiency.

[0073] Based on the same inventive concept, this application also provides a monitoring platform for implementing the compression method and decompression method of the numerical monitoring data involved above. The implementation solution provided by this monitoring platform is similar to the implementation solution described in the above method. Therefore, the specific limitations in the embodiments of the monitoring platform provided below can be referred to the limitations of the compression method and decompression method of the numerical monitoring data above, and will not be repeated here.

[0074] like Figure 5 As shown, the monitoring platform provided in the embodiment of the present application includes an acquisition module 501, a processing module 502, and a memory 503. The processing module 502 can use various interfaces and lines to connect various parts within the entire monitoring platform, and by running or executing instructions, programs, code sets, or instruction sets stored in the memory 503, call the data stored in the memory 503, perform various functions of the monitoring platform, and process data to implement the various method steps provided in the above embodiments. It will be understood by those skilled in the art that the structure of the monitoring platform shown in the figure does not constitute a limitation on the monitoring platform.

[0075] Among them, the acquisition module 501 is at least used to collect numerical monitoring data; the processing module 502 records the numerical monitoring data in the form of an array; the processing module 502 is also used to set a reference value, calculate the deviation between each element in the array and the reference value, and record the coordinates of the elements with non-zero deviations. The reference value is configured to make the proportion of elements with non-zero deviations in the total elements less than a preset proportion; the processing module 502 is configured to generate a compressed data packet, which includes the reference value, the non-zero deviation, and the coordinates of the elements with non-zero deviations.

[0076] In this embodiment, numerical monitoring data is collected by the acquisition module 501, and the numerical monitoring data is recorded in the form of an array by the processing module 502; the processing module 502 is also used to set a reference value, calculate the deviation between each element in the array and the reference value, and record the coordinates of the elements with non-zero deviations. The reference value is configured to make the proportion of elements with non-zero deviations in the total elements less than a preset proportion; the processing module 502 is configured to generate a compressed data packet, which includes the reference value, the non-zero deviation, and the coordinates of the elements with non-zero deviations, thereby achieving a higher data compression ratio, reducing data storage space, ensuring the losslessness of the compression process, and preventing any information from being lost during the compression process, thereby improving the efficiency and security of data compression.

[0077] In one embodiment, the processing module 502 is further configured to determine whether the data type of the data in the array is a telemetry data type, and in response to the data type being a telemetry data type, set the median of each tuple in the array as a reference value of the tuple data.

[0078] In one embodiment, the processing module 502 is also used to determine whether the data type of the data in the array is a remote pulse data type. In response to the data type being a remote pulse data type, the sum of the first element of each tuple in the array and the average growth amount of n-1 times the data in the tuple is set as the baseline value of the nth element of the tuple.

[0079] In one embodiment, the processing module 502 is further configured to record a first-moment reference value, calculate a first deviation between each data in the tuple at the second moment and the first-moment reference value, and count the number of elements whose first deviation is non-zero to obtain a first number of non-zero elements; the processing module 502 calculates a second-moment reference value, calculates a second deviation between each data in the tuple at the second moment and the second-moment reference value, and counts the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; the processing module 502 compares the first number of non-zero elements with the second number of non-zero elements, and in response to the first number of non-zero elements being greater than or equal to the second number of non-zero elements, takes the second-moment reference value as the reference value of the tuple at the second moment, and in response to the first number of non-zero elements being less than the second number of non-zero elements, takes the first-moment reference value as the reference value of the tuple at the second moment.

[0080] In one embodiment, the processing module 502 further uses the decompression method of the numerical monitoring data in the above embodiments to restore the compressed data packet in any of the above embodiments.

[0081] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the claims appended to this application.

Claims

1. A method for compressing numerical monitoring data, characterized in that: The method comprises the following steps: Receiving the numerical monitoring data, and constructing an array, using the numerical monitoring data as an element of the array; Determine whether the data type of the data in the array is a telemetry data type or a telepulse data type, and set corresponding reference values ​​according to the different data types of the elements in the array, wherein, in response to determining that the data type is a telemetry data type, the median of each tuple in the array is set as the reference value of the tuple data; in response to determining that the data type is a telepulse data type, the sum of the first element of each tuple in the array and the average growth amount of n-1 times the data in the tuple is set as the reference value of the nth element of the tuple; Calculating the deviation between each element in the array and the reference value, recording the coordinates of the elements with non-zero deviation, wherein the reference value is configured so that the proportion of the elements with non-zero deviation in the total elements is less than a preset proportion; A compressed data packet is generated, the compressed data packet including the reference value, the non-zero offset, and the coordinates of the elements whose offsets are non-zero.

2. The method for compressing numerical monitoring data according to claim 1, characterized in that: The method further comprises: Recording a reference value at a first moment, calculating a first deviation between each data in a tuple at a second moment and the reference value at the first moment, and counting the number of elements whose first deviation is non-zero to obtain a first number of non-zero elements; Calculating a second-time reference value, calculating a second deviation between each data in the tuple at the second time and the second-time reference value, and counting the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; Compare the first number of non-zero elements with the second number of non-zero elements. In response to the first number of non-zero elements being greater than or equal to the second number of non-zero elements, take the second moment reference value as the reference value of the tuple at the second moment. In response to the first number of non-zero elements being less than the second number of non-zero elements, take the first moment reference value as the reference value of the tuple at the second moment.

3. The method for compressing numerical monitoring data according to claim 1, characterized in that: The method further comprises: Recording a reference value at a first moment, calculating a first deviation between each data in a tuple at a second moment and the reference value at the first moment, and counting the number of elements whose first deviation is non-zero to obtain a first number of non-zero elements; Calculating a second-time reference value, calculating a second deviation between each data in the tuple at the second time and the second-time reference value, and counting the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; Calculate a benchmark mean, where the benchmark mean is obtained by taking a weighted sum of the benchmark value at the first moment and the benchmark value at the second moment, calculate a third deviation between each data in the tuple at the second moment and the benchmark mean, and count the number of elements whose third deviation is non-zero to obtain a third number of non-zero elements; Compare the first number of non-zero elements, the second number of non-zero elements and the third number of non-zero elements, and take the first moment reference value, the second moment reference value and the reference mean value that minimizes the number of non-zero elements as the reference value of the tuple at the second moment.

4. The method for compressing numerical monitoring data according to claim 1, characterized in that: In response to the numerical monitoring data fluctuating within a preset range, the data type is determined to be a telemetry data type; in response to the numerical monitoring data being accumulated in a time series, the data type is determined to be a telepulse data type.

5. A method for decompressing numerical monitoring data, characterized in that: The decompression method is used to restore the compressed data packet generated by the compression method of the numerical monitoring data according to any one of claims 1 to 4, and the decompression method comprises the following steps: Generate restored data array; The coordinates of the elements in the compressed package whose deviation is non-zero are obtained, and the corresponding positions in the restored data array are filled with restored data, wherein the restored data is the sum of the reference value and the deviation, and the remaining positions in the restored data array are filled with the reference value.

6. A monitoring platform, characterized in that: The monitoring platform comprises: A collection module, the collection module is at least used to collect numerical monitoring data; A processing module, the processing module is used to construct an array, and the numerical monitoring data is used as an element of the array; The processing module is further used to determine whether the data type of the element in the array is a telemetry data type or a telepulse data type, and to set corresponding reference values ​​according to the different data types of the elements in the array, wherein, in response to determining that the data type is a telemetry data type, the median of each tuple in the array is set as the reference value of the tuple data; in response to determining that the data type is a telepulse data type, the sum of the first element of each tuple in the array and the average growth amount of n-1 times the data in the tuple is set as the reference value of the nth element of the tuple; The processing module is further used to calculate the deviation between each element in the array and the reference value, and record the coordinates of the elements with non-zero deviation, wherein the reference value is configured so that the proportion of the elements with non-zero deviation in the total elements is less than a preset proportion; The processing module is configured to generate a compressed data packet, wherein the compressed data packet includes the reference value, the non-zero deviation, and the coordinates of the elements whose deviation is non-zero.

7. The monitoring platform according to claim 6, characterized in that: The processing module is further used to record the reference value at the first moment, calculate the first deviation between each data in the tuple at the second moment and the reference value at the first moment, and count the number of elements whose first deviation is non-zero to obtain the first number of non-zero elements; The processing module calculates a second-time reference value, calculates a second deviation between each data in a tuple at the second time and the second-time reference value, and counts the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; The processing module compares the first number of non-zero elements and the second number of non-zero elements, and in response to the first number of non-zero elements being greater than or equal to the second number of non-zero elements, takes the second moment reference value as the reference value of the tuple at the second moment, and in response to the first number of non-zero elements being less than the second number of non-zero elements, takes the first moment reference value as the reference value of the tuple at the second moment.

8. The monitoring platform according to claim 6, characterized in that: The processing module is further used to record the reference value at the first moment, calculate the first deviation between each data in the tuple at the second moment and the reference value at the first moment, and count the number of elements whose first deviation is non-zero to obtain the first number of non-zero elements; The processing module calculates a second-time reference value, calculates a second deviation between each data in a tuple at the second time and the second-time reference value, and counts the number of elements whose second deviation is non-zero to obtain a second number of non-zero elements; The processing module calculates a reference mean, where the reference mean is obtained by weighted summing the reference value at the first moment and the reference value at the second moment, calculates a third deviation between each data in the tuple at the second moment and the reference mean, and counts the number of elements whose third deviation is non-zero to obtain a third number of non-zero elements; The processing module compares the first number of non-zero elements, the second number of non-zero elements and the third number of non-zero elements, and takes the first moment reference value, the second moment reference value and the reference mean value that minimizes the number of non-zero elements as the reference value of the tuple at the second moment.

9. The monitoring platform according to claim 6, characterized in that: The processing module uses the decompression method of numerical monitoring data as described in claim 5 to restore the compressed data packet.

10. The monitoring platform according to claim 6, characterized in that: The monitoring platform includes a storage module, and the storage module includes a hot data buffer area and a cold data buffer area. In the cold data buffer area, the numerical monitoring data is stored in the form of the compressed data packet.

Citation Information

Patent Citations

  • Real-time parallel compression processing method for telemetry data of unmanned airplane

    CN107025462A

  • Matrix processing method and apparatus, and logic circuit

    CN111010883A

  • Simulation test method, electronic equipment and storage medium

    CN111460635A

  • Data compression method and device, electronic equipment and readable storage medium

    CN117873394A

  • Data processing method, path planning device and related equipment

    CN118227937A