Data Compression Method, Device, Electronic Device and Readable Storage Medium

Abstract

The present application provides a data compression method, apparatus, electronic device, and readable storage medium. In the data compression method provided by the present application, first, the data point type of the target data point is determined according to the time difference between the reporting time of the target data point reported to the reporting system and the processing time of the processing system; wherein, the processing system is used to compress the target data point transmitted from the reporting system; then, the target compression window for compressing the target data point is determined according to the data point type of the target data point; and the target data point is compressed through the target compression window. By determining the data point type of the target data point according to the time difference between the reporting time and the processing time of the target data point, and performing data compression on the target data point using different target compression windows based on different data point types, type-based compression of the target data point is achieved, realizing accurate and efficient compression of the target data point.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly, to a data compression method, apparatus, electronic device, and readable storage medium. Background Art

[0002] Currently, in the implementation of lossy compression, usually the look-ahead points and non-look-ahead points are directly compressed and stored using the same compression window. When some data points are look-ahead points and are compressed using the target compression window, and the time difference between the subsequent data points and the current time is not large, it may cause all subsequent points to be judged as non-look-ahead points during compression. Therefore, compression cannot be performed according to the actual situation of the data points, resulting in a poor compression ratio. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of this application is to provide a data compression method, apparatus, electronic device, and readable storage medium, which can improve the compression ratio of lossy compression.

[0004] In a first aspect, the embodiments of this application provide a data compression method, including: determining the data point type of the target data point according to the time difference between the reporting time of the target data point reported by the reporting system and the processing time of the processing system; wherein, the processing system is used to compress the target data points transmitted from the reporting system; determining a target compression window for compressing the target data point according to the data point type of the target data point; and compressing the target data point through the target compression window.

[0005] In the above implementation process, by determining the data point type of the target data point according to the time difference between the reporting time and the processing time of the target data point, and performing data compression on the target data point using different target compression windows based on different data point types, type-based compression of the target data point is realized, achieving accurate and efficient compression of the target data point. Moreover, the data compression between different types of data points does not affect each other, improving the compression ratio of the target data point.

[0006] In an embodiment, the target compression window includes a fixed compression window; determining the data point type of the target data point according to the time difference between the reporting time of the target data point reported by the reporting system and the processing time of the processing system includes: comparing the time difference with a preset time interval; wherein, the preset time interval is the time interval of the data points in the fixed compression window where the reporting time is ahead of the processing time; determining the data point type of the target data point according to the comparison result.

[0007] In the above implementation process, the data point type of the target data point is determined by comparing the time difference of the target data point with the preset time interval, so as to determine whether there is an error in the target data point during the upload process, and to distinguish normal data points from abnormal data points, thereby improving the classification accuracy of the target data point.

[0008] In one embodiment, the data point types include early points and non-early points; determining the data point type of the target data point according to the comparison result includes: if the comparison result is that the time difference is greater than the preset time interval, determining that the target data point is an early point; if the comparison result is that the time difference is not greater than the preset time interval, determining that the target data point is a non-early point.

[0009] In the above implementation process, by comparing the time difference of the target data point with the preset time interval, it is determined whether the target data point is an early point or a non-early point, and further whether there is an abnormality in the system during the transmission of the target data point, realizing the determination of the system situation through the data point type of the target data point, and improving the monitoring ability of the system.

[0010] In one embodiment, the target compression window includes a temporary compression window and a fixed compression window; wherein, the temporary compression window is used to compress the current batch of data points it receives, and the fixed compression window is used to compress multiple batches of data points; the multiple batches of data points include the data points of the current batch received by the fixed compression window and one or more batches of data points after the current batch; determining the target compression window for compressing the target data point according to the data point type of the target data point includes: if the target data point is an early point, determining that the target compression window is a temporary compression window; if the target data point is a non-early point, determining that the target compression window is a fixed compression window.

[0011] In the above implementation process, early points are compressed by the temporary compression window, and non-early points are compressed by the fixed compression window. When the current batch of data points is an early point, the current batch of data points is compressed by the temporary compression window. When the next batch of data points is a non-early point, the next batch of data points is compressed by the fixed compression window, preventing the compression of the current batch of early points from affecting the compression of subsequent non-early point data and improving the compression rate of data compression.

[0012] In one embodiment, the target data points include leading points; wherein, the leading points are target data points with a time difference greater than a preset time interval; the preset time interval is the time interval of the reported time relative to the processing time for the data points leading in a fixed compression window; the fixed compression window is used to compress multiple batches of data points; after compressing the target data points through the target compression window, the method further includes: updating the leading point ratio of the processing system according to the data point type of the target data points; updating the target compression window of the leading points according to the updated leading point ratio.

[0013] In the above implementation process, by predicting the reason for the formation of leading points according to the leading point ratio, and then updating the target compression window of the leading points, the target compression window of the leading points can be updated and changed accordingly according to the actual generation reason of the leading points, so that the target compression window of the leading points is more in line with the actual situation of data point compression of the target data points, to improve the flexibility of target data point compression, and then improve the compression accuracy rate.

[0014] In one embodiment, the target data points further include non-leading points; wherein, the non-leading points are target data points with a time difference not greater than the preset time interval; the updating the target compression window of the leading points according to the updated leading point ratio of the processing system includes: comparing the updated leading point ratio of the processing system with a ratio threshold, and if the updated leading point ratio of the processing system is greater than the ratio threshold, updating the compression window of the leading points to the compression window of the non-leading points.

[0015] In the above implementation process, when the leading point ratio is greater than the threshold, it can be determined that the leading point is caused by system error. In order to reduce the inaccurate compression of the data points earlier due to system error, by updating the target compression window of the leading point to the fixed compression window, regarding the leading point as a non-leading point, ignoring the influence of system error on the compression rate, improving the accuracy rate of data compression and ensuring a good compression rate of the target data points.

[0016] In one embodiment, the updating the leading point ratio of the processing system according to the data point type of the target data points includes: respectively obtaining the current total number of data points and the current number of leading points in multiple time windows, where the time window is the window between the startup time point and the set time point of the processing system; calculating the leading point ratios in multiple time windows respectively through the current total number of data points and the current number of leading points; updating the leading point ratio of the processing system according to the leading point ratios in multiple time windows.

[0017] In the above implementation process, the percentage of leading points of the processing system is calculated based on the percentage of leading points in multiple time windows, so that the percentage of leading points of the processing system is comprehensively calculated from multiple time windows, ensuring the association between the calculation of the percentage of leading points and the time windows, making the calculation of the percentage of leading points more in line with the actual situation of the processing system, and improving the accuracy of the calculation of the percentage of leading points.

[0018] In a second aspect, an embodiment of the present application further provides a data compression device, including: a first determination module: configured to determine the data point type of the target data point according to the time difference between the reporting time of the target data point reporting system and the processing time of the processing system; wherein, the processing system is used to compress the target data point transmitted from the reporting system; a second determination module: configured to determine a target compression window for compressing the target data point according to the data point type of the target data point; a compression module: configured to compress the target data point through the target compression window.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor and a memory, the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, when the machine-readable instructions are executed by the processor, the steps of the method in the above first aspect, or any possible implementation manner of the first aspect are executed.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the data compression method in the above first aspect, or any possible implementation manner of the first aspect are executed.

[0021] To make the above objects, features, and advantages of the present application more obvious and understandable, specific embodiments are hereinafter given, and in conjunction with the accompanying drawings, the following detailed description is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 Schematic diagram of determining leading points in the prior art;

[0024] Figure 2 Schematic diagram of the interaction between the reporting system and the processing system provided by the embodiment of the present application;

[0025] Figure 3 Block diagram of the electronic device provided by the embodiment of the present application;

[0026] Figure 4 Flowchart of the data compression method provided by the embodiment of the present application;

[0027] Figure 5 Flowchart of the data prediction method and corresponding processing method in data compression provided by the embodiment of the present application;

[0028] Figure 6 Schematic diagram of the functional modules of the data compression device provided by the embodiment of the present application. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0030] It should be noted that: Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0031] In a database, generally, a large amount of data is reported by devices, but in actual use, not all the data is required, only the key information is needed. Therefore, in the storage process, a lossy compression algorithm is generally used, only a small number of data points are retained, and the data at other time points can be restored by linear interpolation. For ease of understanding, the following takes the pi rotation door compression algorithm as an example to describe the data compression process, which is specifically as follows:

[0032] If there is an advanced point in the current batch, the first advanced point A in the current batch is used as the reference point, and the last point D in the current batch is used as the last point. According to the data value and numerical range where the last point is located, the two points above and below the last point are determined. Specifically, assuming the data range is d and the data value where the last point is located is a, the two points above and below the last point are a±d of the data value where the last point is located. After determining the two points above and below the last point, the two points above and below the last point are respectively connected to the reference point A to determine the effective area, and all the data in the current batch is compressed and stored. When the data of the next batch is uploaded, it is judged whether the newly uploaded data point is within the range of the effective area. If the data point is within the range of the effective area, the data point is stored. If the data point is not within the range of the effective area, the last point is stored in the fixed storage, and the last point is used as the reference point for a new round of compression.

[0033] Exemplarily, as Figure 1 shownFigure 1 Point A in it is the reference point, and point D is the last point. The area between the two connecting lines in the coordinates in the figure is the effective area range. At this time, it can be judged that point E is within the effective area range, so this point E is stored. Point F is outside the effective area range, so this point D is stored, and point D is used as the reference point for the new round of compression. The effective area range is determined again according to the above method, and the subsequent data points are judged. During the above compression process, the data points are uploaded in ascending order of time. When there are leading points, since the reference point will be re-determined with the appearance of the leading points, when the time of some data points is incorrect, and the time difference between the time of the subsequent data points and the current time is not large, it will cause all the subsequent points to become non-leading points, and the compression cannot be performed according to the actual situation of the data points, resulting in a poor compression rate.

[0034] In view of this, the inventor of the present application proposes a data compression method. By determining the data point type of the data point according to the relationship between the reporting time and the processing time of the data point, the compression window for compressing the data point is determined based on the data point type of the data point, and then the data point is compressed to prevent the compression between different types of data points from affecting each other, so as to improve the compression rate of the data point compression.

[0035] To facilitate the understanding of this embodiment, first, the operating environment for implementing a data compression method disclosed in the embodiments of the present application is introduced in detail.

[0036] As Figure 2 shown, it is a schematic diagram of the interaction between the reporting system 200 and the processing system 300 provided by the embodiment of the present application. The processing system 300 is communicatively connected to one or more reporting systems 200 through a network for data communication or interaction. The processing system 300 can be a network server, a database server, etc., and can also be a personal computer (PC), a tablet computer, a smart phone, a personal digital assistant (PDA), etc. The reporting system 200 can be a personal computer (PC), a tablet computer, a smart phone, a personal digital assistant (PDA), etc.

[0037] Optionally, the reporting system 200 and the processing system 300 can be two different devices in the same electronic device, or two different electronic devices, or the same electronic device. The specific devices of the reporting system 200 and the processing system 300 can be selected according to the actual situation, and the present application does not make specific limitations.

[0038] The reporting system 200 here is used to upload its own data or other acquired data to the processing system 300 and generate a reporting time.

[0039] The processing system 300 here is used to acquire the data uploaded by the reporting system 200 and perform compression processing on the data. The processing system 300 is also used to execute the data compression method disclosed in this embodiment.

[0040] For ease of understanding of this embodiment, the electronic device that executes the data compression method disclosed in the embodiments of the present application will be introduced in detail below.

[0041] It can be understood that the electronic device in the embodiments of the present application may be a processing system.

[0042] As Figure 3 shown, it is a block diagram of the electronic device. The electronic device 100 may include a memory 111, a storage controller 112, a processor 113, and a peripheral interface 114. Those of ordinary skill in the art can understand that Figure 3 the structure shown is only illustrative and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include more or fewer components than Figure 3 shown, or have a different configuration from Figure 3 shown.

[0043] The above-mentioned memory 111, storage controller 112, processor 113, and peripheral interface 114 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other through one or more communication buses or signal lines. The above-mentioned processor 113 is used to execute the executable module stored in the memory.

[0044] Among them, the memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory 111 is used to store programs. After receiving an execution instruction, the processor 113 executes the program. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of this application can be applied to or implemented by the processor 113.

[0045] The above-mentioned processor 113 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0046] The above-mentioned peripheral interface 114 couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 may be implemented on a single chip. In other instances, they may be implemented by separate chips respectively.

[0047] The electronic device 100 in this embodiment can be used to execute each step in the various methods provided in the embodiments of this application. The implementation process of the data compression method will be described in detail through several embodiments below.

[0048] Please refer to Figure 4 , which is a flowchart of the data compression method provided by the embodiments of this application. The following will be forFigure 4 The specific process shown will be elaborated in detail.

[0049] Step 201: Determine the data point type of the target data point according to the time difference between the reporting time of the target data point reported by the reporting system and the processing time of the processing system.

[0050] Among them, the processing system is used to compress the target data point transmitted from the reporting system. The target data point is the data point to be compressed.

[0051] The data point types here include leading points and non-leading points.

[0052] The above reporting time can be the time when the reporting system uploads the target data point, or the time set through settings. The reporting time can be uploaded together with the target data point, or uploaded separately. The uploading method of the reporting time can be custom-set, and this application does not make specific restrictions.

[0053] The reporting system here is used to upload its own data or other data obtained to the processing system and generate a reporting time. The other data can be data sent by other devices to the reporting system, or data to be input into the reporting system. The other data can be selected or set according to the actual situation, and this application does not make specific restrictions.

[0054] The above processing time is the time when the processing system compresses the target data point.

[0055] It can be understood that when the reporting system reports the target data point, it can report in batches, and one batch can include multiple target data points.

[0056] Step 202: Determine the target compression window for compressing the target data point according to the data point type of the target data point.

[0057] The target window here can include a temporary compression window and a fixed compression window. Among them, the temporary compression window is used to compress the current batch of data points it receives. The fixed compression window is used to compress multiple batches of data points, and the multiple batches of data points include the current batch of data points received by the fixed compression window and one or more batches of data points after the current batch.

[0058] Step 203: Compress the target data point through the target compression window.

[0059] In the above implementation process, the data point type of the target data point is determined according to the time difference between the reporting time and the processing time of the target data point, and the target data point is compressed based on different target compression windows used for different data point types, so as to compress the target data point by type, achieving accurate and efficient compression of the target data point. Moreover, the data compression between different types of data points does not affect each other, improving the compression rate of the target data point.

[0060] In a possible implementation manner, step 201 includes: comparing the time difference with a preset time interval; determining the data point type of the target data point according to the comparison result.

[0061] Wherein, the preset time interval is the time interval of the data point whose reporting time is ahead of the processing time in the fixed compression window.

[0062] In some embodiments, the preset time interval can also be a fixed value, such as the time interval is 1s, 1min, 1h, etc. If the preset time interval is a fixed value, the preset time interval can be set manually or obtained by calculation of an electronic device, and the setting method of the preset time interval can be set according to the actual situation, and the present application does not make specific limitations.

[0063] In the above implementation process, the data point type of the target data point is determined according to the comparison result between the time difference of the target data point and the preset time interval, so as to determine whether an error occurs during the upload of the target data point, and to distinguish normal data points from abnormal data points, improving the classification accuracy of the target data point.

[0064] In a possible implementation manner, determining the data point type of the target data point according to the comparison result includes: if the comparison result is that the time difference is greater than the preset time interval, determining that the target data point is an ahead point; if the comparison result is that the time difference is not greater than the preset time interval, determining that the target data point is a non-ahead point.

[0065] In some embodiments, the reporting time of the target data point can also be directly compared with the processing time. If the reporting time of the target data point is later than the processing time, it is determined that the target data point is an advanced point. If the reporting time of the target data point is earlier than the processing time, it is determined that the target data point is a non-advanced point. For example, if the reporting time of the target data point is 14:00 and the processing time of the target data point is 13:00, it indicates that there may be an abnormality in the transmission of the target data point, resulting in the reporting time of the target data point being greater than the processing time. At this time, it is determined that the target data point is an advanced point. Similarly, if the time interval is not greater than the preset time interval, it indicates that the reporting time of the target data point is not greater than the processing time, and there is no abnormality in the transmission of the target data point. For example, if the reporting time of the target data point is 13:00 and the processing time of the target data point is 14:00, it is determined that the target data point is a non-advanced point at this time.

[0066] In the above implementation process, by comparing the time difference of the target data point with the preset time interval, it is determined whether the target data point is an advanced point or a non-advanced point, and further whether there is an abnormality in the system during the transmission of the target data point, realizing the determination of the system situation through the data point type of the target data point, and improving the monitoring ability of the system.

[0067] In a possible implementation manner, step 202 includes: if the target data point is an advanced point, determining the target compression window as a temporary compression window; if the target data point is a non-advanced point, determining the target compression window as a fixed compression window.

[0068] If all the data points in the current batch and subsequent batches are non-advanced points, the fixed compression window is always used. When it is determined that a batch of data points is an advanced point, the temporary window is used for compression, and only the target data points in this batch are compressed. If the data points in the batches after this batch are non-advanced points, the fixed compression window is continued to be used.

[0069] It can be understood that the fixed compression window can compress multiple batches of data at the same time. For example, the reporting system reports 3 batches of data points to the processing window, which are batch 1, batch 2, and batch 3 in chronological order, and the data points in batch 1, batch 2, and batch 3 are not advanced points. Then the fixed compression window can compress the data in batch 1, batch 2, and batch 3 at the same time, and calculate the compression ratio according to all the data points in batch 1, batch 2, and batch 3. By compressing the data points in batch 1, batch 2, and batch 3 together and calculating the compression ratio, the compression between multiple batches of data points can be balanced to prevent the problem of low compression ratio accuracy caused by a large difference in the number of data points between batches.

[0070] In the above implementation process, by compressing the leading points in the window during temporary compression and using a fixed compression window to compress non-leading points, when the current batch of data points is a leading point, the current batch of data points is compressed using a temporary compression window. When the next batch of data points is a non-leading point, the next batch of data points is compressed using a fixed compression window, preventing the compression of the current batch of leading points from affecting the subsequent compression of non-leading point data and improving the compression ratio of data compression.

[0071] In a possible implementation manner, after step 203, the data compression method further includes: updating the leading point proportion of the processing system according to the data point type of the target data point; updating the target compression window of the leading point according to the updated leading point proportion.

[0072] When the data point type of the target data point is a leading point, it indicates that the number of leading points in the processing system has changed. Therefore, it is necessary to calculate the leading point proportion according to the current number of leading points and the total number of data points in the processing system to update the leading point proportion of the processing system.

[0073] It can be understood that when the leading point proportion is relatively large, it indicates that there are more leading points in the current processing system. According to this leading point proportion, the reason for the appearance of the leading point can be predicted. If the leading point proportion is within the normal range, it indicates that the leading point is caused by the data point transmission process. If the leading point proportion is too large and exceeds the normal range, it indicates that the leading point may be caused by the error between the reporting system and the processing system itself. Further, the target data point can be compressed using the corresponding target compression window according to the formation reason of the leading point.

[0074] If the leading point is caused by the system error between the upload system and the processing system, since there is an error between the systems themselves, the subsequent input data points may also be leading points. At this time, a temporary compression window is used to compress the data points, resulting in a low compression ratio for each batch of data points, just like the leading points, and further causing the compression ratio of all batches of data points to be inaccurate. If the target compression window is replaced with a fixed compression window at this time, the leading points caused by the system error are treated as non-leading points, so that all subsequent batches of data points are compressed according to the fixed compression window, thus ensuring the compression ratio of data point compression under the system error.

[0075] In the above implementation process, by predicting the formation reason of the leading point according to the leading point proportion, and then updating the target compression window of the leading point, the target compression window of the leading point can be updated and changed accordingly according to the actual generation reason of the leading point, making the target compression window of the leading point more in line with the actual situation of the target data point during data point compression, so as to improve the flexibility of the target data point compression and further improve the compression accuracy.

[0076] In a possible implementation, updating the leading point ratio of the processing system according to the data point type of the target data point includes: comparing the updated leading point ratio of the processing system with a ratio threshold. If the updated leading point ratio of the processing system is greater than the ratio threshold, updating the compression window of the leading point to the compression window of the non-leading point.

[0077] Understandably, if the leading point ratio of the processing system exceeds the threshold, it indicates that the generation of the leading point may not be caused by the data point transmission process, but by the system error or other errors between the upload system and the processing system itself. Then it can be predicted that most of the subsequent data points may be leading points. At this time, to avoid the reduction of the compression efficiency of the system, the system can be adjusted, that is, the leading point is identified as a non-leading point, and the target compression window of the leading point is updated to a fixed compression window.

[0078] In some embodiments, to prevent misjudgment of the data point type of the target data point due to the error of the system itself, when it is determined that the leading point ratio of the processing system exceeds the threshold, the preset time interval can be further updated according to the reporting time and processing time of the leading point, and then whether the subsequent target data points are leading points is judged according to the updated preset time interval.

[0079] Optionally, updating the preset time interval according to the reporting time and processing time of the leading point can be: the preset time interval is equal to the difference between the reporting time and the processing time of the leading point.

[0080] Understandably, as Figure 5 shown, Figure 5 is a flowchart of the data prediction method and the corresponding processing method in data compression provided by the embodiment of the present application. When the processing system obtains the target data point, the time difference of the target data point is compared with the preset time threshold. If the time difference is greater than the preset time threshold, it is determined that the target data point is a leading point, and the leading point is compressed using a temporary compression window. If the time difference is not greater than the preset time threshold, it is determined that the target data point is a non-leading point, and the non-leading point is compressed using a fixed compression window. After compressing the target data point, the leading point ratio in the processing system is updated according to the number of leading points and the total number of data points in the processing system, and the updated leading point ratio is compared with the ratio threshold. If the updated leading point ratio of the processing system is greater than the ratio threshold, the compression window of the leading point is updated to the compression window of the non-leading point, and the preset time threshold is updated to the time difference between the reporting time and the processing time of the target data point.

[0081] In the above implementation process, when the proportion of leading points is greater than the threshold, it can be determined that the leading point is caused by system error. In order to reduce the inaccurate compression of early data points due to system error, the target compression window of the leading point is updated to a fixed compression window, so as to regard the leading point as a non-leading point, ignoring the influence of system error on the compression rate, improving the accuracy of data compression and ensuring a good compression rate of target data points.

[0082] In a possible implementation manner, updating the proportion of leading points of the processing system according to the data point type of the target data points includes: respectively obtaining the current total number of data points and the current number of leading points in multiple time windows; calculating the proportion of leading points in multiple time windows respectively through the current total number of data points and the current number of leading points; and updating the proportion of leading points of the processing system according to the proportion of leading points in multiple time windows.

[0083] Among them, the time window is the window between the start time point of the processing system and the set time point. For example, the time window is 10s, 80s, 1280s, etc. The set time can be set and can be adjusted according to the actual situation, and the application does not make specific restrictions.

[0084] The following takes the time windows of 10s, 80s, and 1280s as examples to further illustrate the calculation of the proportion of leading points:

[0085] Obtain the number NS1 of all data points and the number NP1 of leading points in the period from the start time point of the processing system to 10s, and calculate the proportion of leading points p1 = NP1 / NS1 in the 10s window according to the number of all data points and the number of leading points.

[0086] Obtain the number NS2 of all data points and the number NP2 of leading points in the period from the start time point of the processing system to 80s, and calculate the proportion of leading points p2 = NP2 / NS2 in the 80s window according to the number of all data points and the number of leading points.

[0087] Obtain the number NS3 of all data points and the number NP3 of leading points in the period from the start time point of the processing system to 1280s, and calculate the proportion of leading points p3 = NP3 / NS3 in the 80s window according to the number of all data points and the number of leading points.

[0088] Then calculate the proportion of leading points in the processing system according to the proportion of leading points in the above 10s, 80s, and 1280s time windows: p = k1 * p1 + k2 * p2 + k3 * p3. Where k1 is the weight of the 10s time window, k2 is the weight of the 80s time window, and k3 is the weight of the 1280s time window.

[0089] In some embodiments, in the above-mentioned calculation of the leading point ratio, the data in the 10s time window is updated to the 80s time window every 10s, and the data in the 80s time window is updated to the 1280s time window every 80s.

[0090] In the above implementation process, by calculating the leading point ratio of the processing system according to the leading point ratios of multiple time windows, the leading point ratio of the processing system is obtained through comprehensive calculation of multiple time windows, ensuring the association between the leading point ratio calculation and the time window, making the calculation of the leading point ratio more in line with the actual situation of the processing system, and improving the accuracy of the leading point ratio calculation.

[0091] Based on the same inventive concept, the embodiments of the present application also provide a data compression device corresponding to the data compression method. Since the principle of solving problems by the device in the embodiments of the present application is similar to that of the foregoing data compression method embodiments, the implementation of the device in this embodiment can refer to the description in the method embodiments above, and the repeated parts will not be elaborated.

[0092] Please refer to Figure 6 , which is a schematic diagram of the functional modules of the data compression device provided by the embodiments of the present application. Each module in the data compression device in this embodiment is used to execute each step in the above method embodiment. The data compression device includes a first determination module 301, a second determination module 302, and a compression module 303; wherein,

[0093] The first determination module 301 is used to determine the data point type of the target data point according to the time difference between the reporting time of the target data point reporting system and the processing time of the processing system; wherein, the processing system is used to compress the target data points transmitted from the reporting system.

[0094] The second determination module 302 is used to determine the target compression window for compressing the target data point according to the data point type of the target data point.

[0095] The compression module 303 is used to compress the target data point through the target compression window.

[0096] In a possible implementation manner, the first determination module 301 is further used to: compare the time difference with a preset time interval; wherein, the preset time interval is the time interval of the data points in the fixed compression window where the reporting time is ahead of the processing time; determine the data point type of the target data point according to the comparison result.

[0097] In a possible implementation manner, the first determination module 301 is specifically configured to: if the comparison result is that the time difference is greater than the preset time interval, determine that the target data point is an advanced point; if the comparison result is that the time difference is not greater than the preset time interval, determine that the target data point is a non-advanced point.

[0098] In a possible implementation manner, the first determination module 301 is specifically configured to: if the target data point is an advanced point, determine that the target compression window is a temporary compression window; if the target data point is a non-advanced point, determine that the target compression window is a fixed compression window.

[0099] In a possible implementation manner, the data compression device further includes an update module, configured to update the proportion of advanced points of the processing system according to the data point type of the target data point; and update the target compression window of the advanced points according to the updated proportion of advanced points.

[0100] In a possible implementation manner, the update module is specifically configured to compare the updated proportion of advanced points of the processing system with a proportion threshold. If the updated proportion of advanced points of the processing system is greater than the proportion threshold, update the compression window of the advanced points to the compression window of non-advanced points.

[0101] In a possible implementation manner, the update module is specifically configured to respectively obtain the current total number of data points and the current number of advanced points in multiple time windows, where the time window is a window between the startup time point of the processing system and the set time point; calculate the proportion of advanced points in multiple time windows respectively through the current total number of data points and the current number of advanced points; and update the proportion of advanced points of the processing system according to the proportion of advanced points in multiple time windows.

[0102] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the data compression method described in the foregoing method embodiment.

[0103] The computer program product of the data compression method provided by the embodiment of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the data compression method described in the foregoing method embodiment. For details, refer to the foregoing method embodiment, and details are not described herein again.

[0104] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the part of the module, program segment, or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0105] In addition, each functional module in various embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0106] When the above-mentioned functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs. It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the elements defined by the statement "including..." do not exclude the existence of additional identical elements in the process, method, article or device including the said elements.

[0107] The foregoing are only the preferred embodiments of this application and are not used to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application shall be included within the protection scope of this application. It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0108] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by this application, and all should be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

1. A data compression method, characterized in that, comprising: determining the data point type of the target data point according to the time difference between the reporting time of the target data point reporting system and the processing time of the processing system; wherein, the processing system is used to compress the target data points transmitted from the reporting system; determining a target compression window for compressing the target data point according to the data point type of the target data point; and compressing the target data point through the target compression window.

2. The method according to claim 1, characterized in that, wherein, the target compression window includes a fixed compression window; the determining the data point type of the target data point according to the time difference between the reporting time of the target data point reporting system and the processing time of the processing system includes: comparing the time difference with a preset time interval; wherein, the preset time interval is the time interval of the data points in the fixed compression window where the reporting time is ahead of the processing time; determining the data point type of the target data point according to the comparison result.

3. The method according to claim 2, characterized in that, wherein, the data point types include advanced points and non-advanced points; the determining the data point type of the target data point according to the comparison result includes: if the comparison result is that the time difference is greater than the preset time interval, determining that the target data point is an advanced point; if the comparison result is that the time difference is not greater than the preset time interval, determining that the target data point is a non-advanced point.

4. The method according to claim 3, characterized in that, wherein, the target compression window includes a temporary compression window and a fixed compression window; wherein, the temporary compression window is used to compress the current batch of data points it receives, and the fixed compression window is used to compress multiple batches of data points; the multiple batches of data points include the data points of the current batch received by the fixed compression window and one or more batches of data points after the current batch; the determining a target compression window for compressing the target data point according to the data point type of the target data point includes: if the target data point is an advanced point, determining that the target compression window is a temporary compression window; if the target data point is a non-advanced point, determining that the target compression window is a fixed compression window.

5. The method according to claim 1, characterized in that, wherein, the target data points include advanced points; wherein, the advanced points are the target data points with a time difference greater than the preset time interval; the preset time interval is the time interval of the data points in the fixed compression window where the reporting time is ahead of the processing time; the fixed compression window is used to compress multiple batches of data points; after compressing the target data point through the target compression window, the method further includes: updating the proportion of advanced points in the processing system according to the data point type of the target data point; updating the target compression window of the advanced points according to the updated proportion of advanced points.

6. The method according to claim 5, characterized in that, The target data points further include non-advanced points; wherein, the non-advanced points are target data points whose time difference is not greater than the preset time interval; the updating of the target compression window of the advanced points according to the proportion of the advanced points of the updated processing system includes: Comparing the proportion of the advanced points of the updated processing system with the proportion threshold. If the proportion of the advanced points of the updated processing system is greater than the proportion threshold, updating the compression window of the advanced points to the compression window of the non-advanced points.

7. The method according to claim 5, wherein, the updating of the proportion of the advanced points of the processing system according to the data point type of the target data points includes: respectively obtaining the current total number of data points and the current number of advanced points in a plurality of time windows, where the time window is the window between the start time point of the processing system and the set time point; calculating the proportion of the advanced points in a plurality of the time windows respectively through the current total number of data points and the current number of advanced points; updating the proportion of the advanced points of the processing system according to the proportion of the advanced points in a plurality of the time windows.

8. A data compression device, wherein, comprising: A first determination module: configured to determine the data point type of the target data points according to the time difference between the reporting time of the target data points reported by the reporting system and the processing time of the processing system; wherein, the processing system is configured to compress the target data points transmitted from the reporting system; A second determination module: configured to determine a target compression window for compressing the target data points according to the data point type of the target data points; A compression module: configured to compress the target data points through the target compression window.

9. An electronic device, wherein, comprising: A processor and a memory, the memory stores machine-readable instructions executable by the processor. When the electronic device runs, when the machine-readable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are executed.

10. A computer-readable storage medium, wherein, a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images