A data compression method, apparatus, electronic device, and storage medium

Abstract

This invention provides a data compression method, apparatus, electronic device, and storage medium. The method includes: segmenting the original data using a breakpoint detection algorithm; fitting each segment of data using a fitting algorithm; and obtaining the restored data using a decompression algorithm. This addresses the problem of excessively large data volumes and high storage and transmission costs in current massive data acquisition scenarios, thus promoting the further development of various data platforms for digital transformation in the big data era. The method of segmenting the original data using a breakpoint detection algorithm and fitting each segment of data using a fitting algorithm effectively achieves efficient data compression. The restoration of the data through a decompression algorithm further addresses the problem of excessively large data volumes and high storage and transmission costs in current massive data acquisition scenarios, thus promoting the further development of various data platforms for digital transformation in the big data era.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data compression method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of IoT-related technologies, smart manufacturing has grown rapidly in recent years and has a very broad prospect. However, the problem of massive data volumes and difficulty in retaining them in databases is becoming increasingly serious. In order to meet the requirements of storing and re-accessing ultra-large-scale data, there is an urgent need for a more effective data compression and decompression method and device.

[0003] Currently, most column-oriented databases commonly use three dedicated compression algorithms: Run-Length Encoding, Bit-Vector Encoding, and Dictionary Encoding. Besides these three, general-purpose compression algorithms such as Snappy, zlib, and LZO are also widely used in column-oriented databases, often applying multiple compression algorithms to the same column. However, for the massive data volumes faced by the Internet of Things and smart manufacturing, the compression efficiency of existing algorithms is insufficient to meet the demands.

[0004] Therefore, how to better achieve data compression has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] This invention provides a data compression method, apparatus, electronic device, and storage medium to solve the problem that data compression cannot be effectively achieved in the prior art.

[0006] This invention provides a data compression method, comprising:

[0007] The original file data is processed by a breakpoint detection algorithm to obtain breakpoint data, and the original file data is segmented based on the breakpoint data to obtain interval data;

[0008] Based on the degree of deviation between each adjacent interval data and in combination with the data compression level, the interval data is adjusted to obtain the adjusted interval data;

[0009] By using a fitting algorithm, the adjusted interval data is fitted according to a preset compression precision to obtain compressed data.

[0010] Based on the data compression method provided by the present invention, after the step of obtaining compressed data, the method further includes:

[0011] Based on the decompression data range information, the target decompression information is determined in the compressed data;

[0012] The target decompression information is decompressed using a decompression algorithm to obtain the decompressed file data.

[0013] According to the data compression method provided by this invention, the step of adjusting the interval data based on the deviation between adjacent interval data and the data compression level specifically includes:

[0014] The similarity and coherence of the data in each interval are analyzed to determine the degree of deviation between adjacent intervals.

[0015] Based on the degree of deviation and data compression level of each adjacent interval, the interval size of each interval is adjusted to obtain the adjusted interval data.

[0016] According to the data compression method provided by the present invention, the step of determining target decompression information in the compressed data based on decompression data interval information specifically includes:

[0017] The target interval information is obtained by comparing the decompressed data interval information with the compressed data interval information;

[0018] The target decompression information is determined from the compressed data based on the target interval information.

[0019] The present invention also provides a data compression apparatus, comprising:

[0020] The segmentation module is used to process the original file data using a breakpoint detection algorithm to obtain breakpoint data, and then segment the original file data according to the breakpoint data to obtain interval data.

[0021] The adjustment module is used to adjust the interval data according to the degree of deviation between each adjacent interval data and the data compression level, so as to obtain the adjusted interval data.

[0022] The compression module is used to fit the adjusted interval data according to the fitting algorithm and the preset compression accuracy to obtain compressed data.

[0023] According to a data compression apparatus provided by the present invention, the apparatus further includes: a decompression module;

[0024] The decompression module is specifically used to determine the target decompression information in the compressed data based on the decompression data range information; and to decompress the target decompression information using a decompression algorithm to obtain the decompressed file data.

[0025] According to a data compression apparatus provided by the present invention, the adjustment module is specifically used for:

[0026] The similarity and coherence of the data in each interval are analyzed to determine the degree of deviation between adjacent intervals.

[0027] Based on the degree of deviation and data compression level of each adjacent interval, the interval size of each interval is adjusted to obtain the adjusted interval data.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the data compression methods described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the data compression methods described above.

[0030] This invention provides a data compression method, apparatus, electronic device, and storage medium. By segmenting the original data using a breakpoint detection algorithm and fitting each segment of data using a fitting algorithm, efficient data compression can be effectively achieved. The decompression algorithm then yields the restored data. This solves the problem of excessively large data volumes and high storage and transmission costs in current massive data collection scenarios, and promotes the further development of various data platforms for digital transformation in the era of big data. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the data compression method provided by the present invention;

[0033] Figure 2 A schematic diagram of the data compression and decompression model provided by this invention;

[0034] Figure 3 A schematic diagram of the data compression device provided by the present invention;

[0035] Figure 4 A schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Figure 1 This is a flowchart illustrating the data compression method provided by the present invention, as shown below. Figure 1 As shown, it includes:

[0038] Step S1: Process the original file data using a breakpoint detection algorithm to obtain breakpoint data, and segment the original file data according to the breakpoint data to obtain interval data;

[0039] Specifically, the breakpoint detection algorithm described in this invention is used to detect breakpoint information in file data, find breakpoint data in the original file data, and then perform preliminary segmentation of the original file data based on the breakpoint data to obtain interval data.

[0040] The breakpoint detection algorithm described in this invention is used to find discontinuities and endpoints in numerical sequence data. Similar algorithms include search-based and zero-crossing-based edge detection algorithms in image processing, and outlier detection algorithms in data cleaning.

[0041] In this invention, the breakpoint data determined by the breakpoint detection algorithm is effectively used to partition the original data file while ensuring data integrity, which is beneficial for subsequent data compression.

[0042] Step S2: Based on the degree of deviation between each adjacent interval data and in combination with the data compression level, the interval data is adjusted to obtain the adjusted interval data;

[0043] The data compression level described in this invention can be set according to the user's requirements for the compression accuracy. The higher the compression level, the more intervals need to be merged, and the larger the intervals are.

[0044] In determining the interval division of compressed data in this invention, not only is the integrity of the data during compression considered based on the breakpoint data, but the degree of deviation between actual data is also fully considered. According to different data compression level requirements, the interval data is further adjusted so that the final adjusted interval data better meets the user's needs and improves the effectiveness of compression.

[0045] Step S3: According to the fitting algorithm, the adjusted interval data is fitted according to the preset compression accuracy to obtain compressed data.

[0046] The fitting algorithm described in this invention is determined based on the data feature information of the interval data.

[0047] The fitting algorithm described in this invention can be an algorithm that provides a unified calling interface and then obtains the fitting result.

[0048] Specifically, the fitting methods can be as follows: data that conforms to the characteristics of a power function will be fitted to a power function; data that conforms to the characteristics of an exponential function will be fitted to an exponential function; data that conforms to the characteristics of a trigonometric function will be fitted to a trigonometric function; and data that conforms to the characteristics of a logarithmic function, an inverse trigonometric function, and their composite forms will all be fitted to the corresponding function forms.

[0049] In this invention, compression accuracy is defined as the degree of deviation between the data obtained after compression and decompression and the original data.

[0050] The preset compression precision described in this invention can be preset according to the user's needs.

[0051] In this invention, the degree of fit to the adjusted interval data is determined based on a preset compression precision, thereby better meeting the user's compression needs.

[0052] This invention segments the original data using a breakpoint detection algorithm and fits each segment using a fitting algorithm, effectively achieving efficient data compression. A decompression algorithm then yields the restored data, addressing the problem of excessively large data volumes and high storage and transmission costs in current massive data acquisition scenarios. This invention promotes the further development of various data platforms for digital transformation in the big data era.

[0053] Based on any of the above embodiments, after the step of obtaining compressed data, the method further includes:

[0054] Based on the decompression data range information, the target decompression information is determined in the compressed data;

[0055] The target decompression information is decompressed using a decompression algorithm to obtain the decompressed file data.

[0056] Specifically, the decompressed data range information described in this invention refers to the range of data that the user wants to decompress.

[0057] However, the decompression data range information here may not correspond to the data range information during compression. Therefore, the decompression data range information and the compressed data range are compared at this time, and the compressed range to be decompressed is selected. The decompressed compressed range must contain the decompression range specified by the user, and finally the target decompression information is obtained.

[0058] This invention decompresses the target decompressed information using a decompression algorithm. After obtaining the decompressed file data, the decompressed data of adjacent compressed intervals are concatenated. If necessary, the decompressed data is also truncated to remove redundant data.

[0059] This invention effectively decompresses compressed data by re-determining the target decompression information through decompressed data range information.

[0060] Based on any of the above embodiments, the step of adjusting the interval data according to the degree of deviation between adjacent interval data and the data compression level specifically includes:

[0061] The similarity and coherence of the data in each interval are analyzed to determine the degree of deviation between adjacent intervals.

[0062] Based on the degree of deviation and data compression level of each adjacent interval, the interval size of each interval is adjusted to obtain the adjusted interval data.

[0063] Specifically, similarity and coherence analysis is performed on the data in each interval to determine the degree of deviation between adjacent intervals. For example, if there are three intervals A, B, and C, the degree of deviation between A and B is high, and the degree of deviation between B and C is low.

[0064] Then, when the compression level is low, A, B, and C can be fitted separately. When the compression level is high, B and C can be combined, A can be fitted separately, and B and C can be combined for fitting.

[0065] This invention can adjust the size of each interval based on the deviation of data in each adjacent interval and the data compression level, and then compress the data according to the adjusted interval data, which can better meet the user's needs for data compression.

[0066] Specifically, based on the different characteristics of the interval data, the fitting algorithm can adaptively fit different types of functions. For example, based on the degree of conformity between the interval data and functions such as power functions, exponential functions, logarithmic functions, trigonometric functions, inverse trigonometric functions, and their combinations, the fitting algorithm performs corresponding fitting. It also includes performing secondary processing on the results of multiple intervals after fitting different functions to different intervals, and further compressing and storing the results.

[0067] This invention determines the corresponding fitting algorithm by using data feature information, thereby achieving better data compression.

[0068] Figure 2 This is a schematic diagram of the data compression and decompression model provided by the present invention, as shown below. Figure 2As shown, the breakpoint detection algorithm processes the original data, combining it with the user-specified compression level to obtain breakpoint data and interval data. Then, the fitting algorithm, combined with the user-specified compression precision, processes the breakpoint data and interval data to obtain the fitted compressed data, which can be stored for later use. When decompression is required, the decompression algorithm processes the fitted compressed data according to the user's decompression data range requirements to obtain the restored data.

[0069] Figure 3 This is a schematic diagram of the data compression device provided by the present invention, as shown below. Figure 3 As shown, it includes: a segmentation module 310, an adjustment module 320, and a compression module 330; wherein, the segmentation module 310 is used to process the original file data using a breakpoint detection algorithm to obtain breakpoint data, and to segment the original file data according to the breakpoint data to obtain interval data; wherein, the adjustment module 320 is used to adjust the interval data according to the degree of deviation between each adjacent interval data and in combination with the data compression level to obtain adjusted interval data; wherein, the compression module 330 is used to fit the adjusted interval data according to a fitting algorithm and a preset compression precision to obtain compressed data.

[0070] More specifically, the device further includes: a decompression module;

[0071] The decompression module is specifically used to determine the target decompression information in the compressed data based on the decompression data range information; and to decompress the target decompression information using a decompression algorithm to obtain the decompressed file data.

[0072] The adjustment module is specifically used for:

[0073] The similarity and coherence of the data in each interval are analyzed to determine the degree of deviation between adjacent intervals.

[0074] Based on the degree of deviation and data compression level of each adjacent interval, the interval size of each interval is adjusted to obtain the adjusted interval data.

[0075] This invention segments the original data using a breakpoint detection algorithm and fits each segment using a fitting algorithm, effectively achieving efficient data compression. A decompression algorithm then yields the restored data, addressing the problem of excessively large data volumes and high storage and transmission costs in current massive data acquisition scenarios. This invention promotes the further development of various data platforms for digital transformation in the big data era.

[0076] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, communication interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a data compression method. This method includes: segmenting the original data using a breakpoint detection algorithm; fitting each segment of data using a fitting algorithm, which effectively achieves efficient data compression; and obtaining the restored data through a decompression algorithm. This solves the problem of excessively large data volumes and high storage and transmission costs in current massive data acquisition scenarios, and promotes the further development of various data platforms for digital transformation in the big data era.

[0077] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the data compression method provided by the above methods. The method includes: segmenting the original data using a breakpoint detection algorithm, fitting each segment of data according to a fitting algorithm, which can effectively achieve efficient data compression, and obtaining the restored data through a decompression algorithm. This solves the problem of excessively large data scale and excessive storage and transmission overhead in current massive data collection scenarios, and promotes the further development of various data platforms for digital transformation in the era of big data.

[0079] On another front, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the data compression method provided in the above embodiments. The method includes: segmenting the original data using a breakpoint detection algorithm, fitting each segment of data according to a fitting algorithm, which can effectively achieve efficient data compression; and obtaining the restored data through a decompression algorithm. This solves the problem of excessively large data scale and excessive storage and transmission overhead in current massive data collection scenarios, and promotes the further development of various data platforms for digital transformation in the era of big data.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data compression method, characterized in that, include: The original file data is processed by a breakpoint detection algorithm to obtain breakpoint data, and the original file data is segmented based on the breakpoint data to obtain interval data; Based on the degree of deviation between each adjacent interval data and in combination with the data compression level, the interval data is adjusted to obtain the adjusted interval data; By using a fitting algorithm and according to a preset compression precision, the adjusted interval data is fitted to obtain compressed data; The adjusted interval data is fitted to obtain compressed data, specifically including: Based on the different characteristics of the adjusted interval data, the adjusted interval data is adaptively fitted into different types of functions, and then the fitting results of multiple interval data are compressed and stored. The step of adjusting the interval data based on the degree of deviation between adjacent interval data and the data compression level specifically includes: The similarity and coherence of the data in each interval are analyzed to determine the degree of deviation between adjacent intervals. Based on the degree of deviation and data compression level of each adjacent interval, the interval size of each interval is adjusted to obtain the adjusted interval data.

2. The data compression method according to claim 1, characterized in that, After the step of obtaining compressed data, the method further includes: Based on the decompression data range information, the target decompression information is determined in the compressed data; The target decompression information is decompressed using a decompression algorithm to obtain the decompressed file data.

3. The data compression method according to claim 2, characterized in that, The step of determining the target decompression information in the compressed data based on the decompression data interval information specifically includes: The target interval information is obtained by comparing the decompressed data interval information with the compressed data interval information; The target decompression information is determined from the compressed data based on the target interval information.

4. A data compression device, characterized in that, include: The segmentation module is used to process the original file data using a breakpoint detection algorithm to obtain breakpoint data, and then segment the original file data according to the breakpoint data to obtain interval data. The adjustment module is used to adjust the interval data according to the degree of deviation between each adjacent interval data and the data compression level, so as to obtain the adjusted interval data. The compression module is used to fit the adjusted interval data to a preset compression precision using a fitting algorithm to obtain compressed data. Specifically, the adjustment module is used for: Based on the different characteristics of the adjusted interval data, the adjusted interval data is adaptively fitted into different types of functions, and then the fitting results of multiple interval data are compressed and stored. The adjustment module is specifically used for: The similarity and coherence of the data in each interval are analyzed to determine the degree of deviation between adjacent intervals. Based on the degree of deviation and data compression level of each adjacent interval, the interval size of each interval is adjusted to obtain the adjusted interval data.

5. The data compression device according to claim 4, characterized in that, The device further includes: a decompression module; The decompression module is specifically used to determine the target decompression information in the compressed data based on the decompression data range information; and to decompress the target decompression information using a decompression algorithm to obtain the decompressed file data.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data compression method as described in any one of claims 1 to 3.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data compression method as described in any one of claims 1 to 3.

Images