Data compression method and device, nonvolatile storage medium and electronic equipment
By determining the sparsity index of the data to be compressed in data compression and selecting the appropriate transformation domain for compression sampling, the problem of impaired compression rate and restore accuracy caused by fixed transformation domain in the prior art is solved, and more efficient data is achieved Compress and restore.
Patent Information
- Application Number
- CN202411141046.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-26
AI Technical Summary
Existing data compression methods perform compression sampling after converting data to a fixed transformation domain, resulting in impairment of compression rate and restore accuracy, which fails to effectively solve this problem.
By determining the sparsity index of the data to be compressed, selecting the most suitable transformation domain for compression sampling, ensuring that the data has good sparsity in the target transformation domain, thereby reducing the accuracy and reduction rate loss during the compression process.
It realizes the improvement of data recovery accuracy while maintaining high compression rates, and solves the problem of impaired compression and recovery accuracy caused by fixed transform domain compression.
Smart Images

Figure CN119030549A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of electronic digital data processing, and in particular, to a data compression method, device, non-volatile storage medium and electronic equipment. Background Art
[0002] In the related art, when compressing data, a fixed transform domain compression scheme is usually adopted, that is, any data is converted to a certain transform domain before compression sampling. The problem with this method is that the sparseness of the converted data cannot be guaranteed, resulting in a certain loss in compression rate and restoration accuracy of the existing compression methods.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present application provide a data compression method, device, non-volatile storage medium and electronic device to at least solve the technical problem of impaired compression rate and restoration accuracy caused by converting data into a fixed transform domain and then performing compressive sampling in the related art.
[0005] According to one aspect of an embodiment of the present application, a data compression method is provided, including: determining a first sparsity index of time series data to be compressed; determining transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than a first preset index threshold; determining a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains based on the transform coefficients corresponding to the multiple transform domains, and determining a target transform domain in each transform domain based on the second sparsity index and the uniform distribution index; compressing and sampling the data to be compressed based on the target transform domain to obtain compressed data corresponding to the data to be compressed.
[0006] Optionally, determining the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index includes: when the second sparsity index corresponding to each transform domain is not less than the second preset index threshold, determining the transform domain with the largest corresponding uniform distribution index as the target transform domain; when there is a non-zero coefficient ratio less than the second preset index threshold in the second sparsity index corresponding to each transform domain, determining the transform domain with the corresponding non-zero coefficient ratio less than the first preset coefficient threshold as the target transform domain; when there are multiple non-zero coefficient ratios less than the second preset threshold in the second sparsity index corresponding to each transform domain, determining the transform domain with the smallest corresponding non-zero coefficient ratio as the target transform domain.
[0007] Optionally, the first sparsity indicator includes the ratio of non-zero elements in the data to be compressed, the second sparsity indicator corresponding to each transform domain includes the ratio of non-zero coefficients in the transform coefficients corresponding to each transform domain, and the uniform distribution indicator corresponding to each transform domain includes the Gini index.
[0008] Optionally, after determining the ratio of non-zero elements in the data to be compressed to all elements in the data to be compressed, the data compression method also includes: when the first sparsity index is less than a first preset threshold, determining that the domain where the data to be compressed is located is the target transformation domain, wherein the data to be compressed includes time series data, and the domain where the data to be compressed is located includes the time domain.
[0009] Optionally, compressing and sampling the data to be compressed according to the target transform domain includes: converting the data to be compressed into the target transform domain; and compressing and sampling the data to be compressed converted into the target transform domain using an observation matrix.
[0010] Optionally, the transform domain includes at least one of the following: a discrete cosine transform domain, a discrete wavelet transform domain, and a discrete Fourier transform domain.
[0011] Optionally, determining the transform coefficients corresponding to the data to be compressed in each transform domain includes: performing a discrete cosine transform on the compressed data to obtain discrete cosine transform coefficients corresponding to the discrete cosine transform domain; performing a discrete wavelet transform on the compressed data to obtain discrete wavelet transform coefficients corresponding to the discrete wavelet transform domain; performing a discrete Fourier transform on the compressed data to obtain discrete Fourier transform coefficients corresponding to the discrete Fourier transform domain.
[0012] Optionally, the data compression method further includes: retrieving target compressed data corresponding to the retrieval instruction according to the retrieval instruction; determining basis vectors and sparse coefficient vectors of the target compressed data; and decompressing the target compressed data according to the basis vectors and the sparse coefficient vectors.
[0013] According to another aspect of an embodiment of the present application, a data compression device is also provided, including: a first processing module, used to determine a first sparsity index of time series data to be compressed; a second processing module, used to determine the transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than a first preset index threshold; a third processing module, used to determine the second sparsity index and uniform distribution index corresponding to each transform domain in a plurality of transform domains based on the transform coefficients corresponding to the plurality of transform domains, and determine the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index; a fourth processing module, used to determine the second sparsity index and uniform distribution index corresponding to each transform domain in a plurality of transform domains based on the transform coefficients corresponding to the plurality of transform domains, and determine the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index.
[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute a data compression method.
[0015] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the data compression method is executed when the program is run.
[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and the computer program executes a data compression method when executed by a processor.
[0017] In an embodiment of the present application, a first sparsity index of time series data to be compressed is determined; when the first sparsity index is not less than a first preset index threshold, the transform coefficients corresponding to the data to be compressed in multiple transform domains are determined; based on the transform coefficients corresponding to the multiple transform domains, a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains are determined, and a target transform domain is determined in each transform domain based on the second sparsity index and the uniform distribution index; based on the target transform domain, the data to be compressed is compressed and sampled to obtain compressed data corresponding to the data to be compressed, the target compression domain is determined by determining the sparsity index and the uniform distribution index of the data to be compressed, and the data to be compressed is converted to the target compression domain and then compressed, thereby achieving the purpose of ensuring the sparsity of the data after conversion, thereby achieving the technical effect of reducing the loss of compression accuracy and restoration rate caused by the compression process, and further solving the technical problem of the compression rate and restoration accuracy being damaged due to converting data to a fixed transform domain and then compressing and sampling in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0019] Figure 1 It is a structural schematic diagram of a computer terminal (mobile terminal) provided according to an embodiment of the present application;
[0020] Figure 2 is a flow chart of a data compression method provided according to an embodiment of the present application;
[0021] Figure 3 It is a schematic diagram for comparing reconstructed data of data to be compressed in the time domain after being processed in the time domain and reconstructed data after being processed in the DCT domain according to an embodiment of the present application;
[0022] Figure 4 It is a schematic diagram of comparison between reconstructed data and original data of data to be compressed that is not sparse in the time domain and is suitable for DCT processing after being processed in the DCT domain according to an embodiment of the present application;
[0023] Figure 5 It is a schematic diagram of comparison between reconstructed data and original data of data to be compressed which is not sparse in the time domain and is not suitable for DWT processing, after being processed in the DWT domain, provided in an embodiment of the present application;
[0024] Figure 6 It is a schematic diagram of comparison between reconstructed data and original data of data to be compressed which is not sparse in the time domain and is processed in the time domain according to an embodiment of the present application;
[0025] Figure 7 It is a schematic diagram of comparison between reconstructed data and original data of data to be compressed that is not sparse in the time domain and is suitable for DWT processing after being processed in the DWT domain according to an embodiment of the present application;
[0026] Figure 8 is a process schematic diagram of a compression sampling algorithm provided according to an embodiment of the present application;
[0027] Fig. 9 is a flow chart of a compression process provided according to an embodiment of the present application;
[0028] Fig.10 It is a flowchart of a decompression process provided according to an embodiment of the present application;
[0029] Fig.11 It is a structural schematic diagram of a data compression and decompression system provided according to an embodiment of the present application;
[0030] Fig.12 It is a structural schematic diagram of a data compression device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0033] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0034] Time domain: Time domain is a representation method that describes how a signal changes over time. In the time domain, a signal is represented as a function of amplitude or intensity that changes over time. Time domain signals can be either continuous or discrete. For continuous time domain signals, a continuous time variable is usually used to represent the signal's change over time. For discrete time domain signals, a series of discrete time points are used to represent the value of the signal at a specific moment. Time domain analysis can reveal some important characteristics of the signal, such as waveform, period, amplitude, etc. However, analyzing the signal only in the time domain may not fully reveal the frequency characteristics of the signal, and then it is necessary to use frequency domain analysis methods, such as Fourier transform.
[0035] Discrete Cosine Transform (DCT) Domain: Discrete Cosine Transform is an orthogonal transform technology widely used in signal processing and data compression. It converts time domain signals to frequency domain and achieves data compression by retaining only important frequency components. DCT represents the signal as the sum of cosine waves of different frequencies through a set of cosine functions, that is, transforms the time signal to the cosine domain. Since most natural signals show the characteristic of concentrated energy in the frequency domain, DCT can concentrate most of the signal energy on the low-frequency components after transformation. By retaining only these important coefficients and quantizing the remaining coefficients, DCT achieves lossy compression. DCT transform is widely used in image and video compression standards such as JPEG, MPEG, etc., and is suitable for time series data with energy concentrated in low frequencies and can tolerate a certain amount of information loss.
[0036] Discrete Wavelet Transform (DWT) domain: Discrete wavelet transform is a signal processing tool that can decompose time domain signals into components of different frequencies and time scales, that is, transform the time signal into the cosine domain. It can be imagined as observing the signal with a magnifying glass of different magnifications to capture the characteristics of the signal at different levels of detail. DWT transforms the signal by using a special waveform called "wavelet". Wavelet is like a template. By stretching and translating this template, the signal can be analyzed at different time positions and frequency ranges. The results of DWT include approximate components and detail components. The approximate components reflect the overall trend of the signal, and the detail components reflect the local details of the signal. It is widely used in speech recognition, image processing, biomedicine and other fields. It is suitable for time series data that contains multi-scale features and local mutations and needs to retain detail information.
[0037] Discrete Fourier Transform (DFT) Domain: Discrete Fourier Transform is a mathematical transformation technique widely used in signal processing and spectrum analysis. It converts the time domain signal to the frequency domain and reveals the spectral characteristics of the signal by representing the frequency components of the signal. DFT represents the signal as the sum of the complex forms of sine and cosine waves of different frequencies through a set of complex exponential functions, that is, transforming the time signal to the frequency domain. In the DFT domain, the amplitude and phase information of the signal are clearly represented. By analyzing the spectrum after DFT transformation, the main frequency components in the signal can be identified, frequency selective filtering can be performed, and frequency domain analysis and processing of the signal can be realized. DFT transformation has important applications in many fields, such as audio processing, radar signal processing, communication systems, etc. DFT can also be used for signal compression and noise reduction by retaining only important frequency components and ignoring or quantizing the remaining components.
[0038] Compressed sampling (CS): Compressed sampling is a novel signal acquisition and compression method that utilizes the sparse nature of the signal. Sparsity means that in a certain transform domain (such as the DWT domain or DCT domain), most of the coefficients are zero or close to zero, and only a few coefficients have significant non-zero values. Traditional signal acquisition follows the Nyquist sampling theorem, and the sampling rate must be greater than twice the highest frequency of the signal. Compressed sampling breaks this limitation. It collects signals at a lower rate, greatly reducing the number of sampling points. Compression is thus achieved. Its advantage is that it greatly reduces the cost of signal acquisition and transmission, and has important application value in bandwidth-limited and energy-limited scenarios. Secondly, it and existing compression methods are at different levels of signal processing, so they can be combined to further compress data. However, existing data compression methods cannot be used in combination.
[0039] Non-zero coefficient ratio: This ratio measures the proportion of non-zero coefficients in the transformed time series data. After the time series data is transformed into the DWT or DCT domain, if most of the coefficients are zero or close to zero, the non-zero coefficient ratio will be very small, indicating that the transformed data has good sparsity.
[0040] Gini index: This index measures the unevenness of the energy distribution of the transformation coefficients. For time series data, if the energy after transformation is concentrated in a few large coefficients, while the energy of other coefficients is very small, the Gini index will be large. This indicates that in the transformed data, the important information is mainly concentrated in a few coefficients, which means that the data is highly sparse.
[0041] Orthogonal Matching Pursuit (OMP): Orthogonal Matching Pursuit is a reconstruction algorithm for solving compressed sampling problems. It is a type of greedy pursuit algorithm. It gradually identifies the sparse support set of the signal through iteration, thereby restoring the original signal. It has high reconstruction accuracy and moderate computational complexity, and is widely used in compressed sampling signal reconstruction tasks.
[0042] With the rapid growth of the number of devices and the rapid development of sensor technology, the amount of network operation data generated is showing an explosive growth trend. These time series data carry key time series information, such as device alarm data, device performance data, device status data, etc., which are indispensable for monitoring system performance, conducting accurate prediction analysis, and implementing efficient real-time control. However, as time series data is continuously generated, how to efficiently store this data has become an urgent issue.
[0043] At present, when storing time series data generated during network operation, there are problems such as limited storage space, high cost, and high resource consumption. Specifically, the time series database storage system faces a hard limit on disk capacity. Especially in the full-professional cloud network environment, the number of time series data points generated by devices every day has reached tens of billions, which puts extremely high requirements on the compression efficiency of time series data. Storing full-professional device performance data not only requires expensive hardware support, such as large-capacity storage devices and data centers, but also data backup and redundancy strategies bring additional cost burdens. In addition, the normal operation of the database requires a large amount of CPU, memory and other resources, especially on the servers that handle data storage and query. Not only does it increase the operating costs of the enterprise, it may also have a certain negative impact on the environment.
[0044] Too much data will also be limited by bandwidth when it is transmitted. Specifically, the continuous transmission of time-series data places a huge demand on bandwidth resources. In scenarios with limited bandwidth (such as remote nodes or mobile devices), continuous data upload may cause network congestion, thus affecting the normal operation of other services.
[0045] In order to solve the above problems, a variety of data compression methods are provided in the relevant technology. The current traditional compression algorithms, such as ZIP, FLAC and LZ series algorithms, need to first obtain the complete original sampling data according to the Nyquist sampling rate, and then generate a 01bit string through quantization coding, divide the bit string into blocks, and use dictionary construction and other means to store the same bit blocks in the bit string without duplication, so as to achieve data compression effect. Due to its wide applicability, traditional compression algorithms have been widely used in file compression, data backup, streaming media transmission and other storage and transmission scenarios involving various types of data (such as text, images, audio, video, etc.), effectively reducing the storage space and transmission bandwidth occupied.
[0046] In contrast, the compression sampling algorithm uses the sparsity of the signal in different transform domains to sample the original data at a rate lower than the Nyquist sampling rate during the sampling phase, directly reducing the number of data points. This unique compression principle enables it to perform well in processing signals with sparse representations, such as equipment monitoring and business time series data signals, because these signals show obvious sparsity in certain transform domains (such as wavelet domain and Fourier domain).
[0047] It can be seen that the traditional compression algorithm compresses the data after full-rate sampling and quantization, while the compressed sampling method directly reduces the sampling points during sampling. Therefore, compressed sampling and traditional methods solve the problem at different levels and are not mutually exclusive. They can be used in series to achieve better compression effects. Taking wireless sensor networks as an example, the compressed sampling algorithm can be used to collect and compress the data first, and then the traditional compression algorithm can be used to compress the compressed data again during the transmission or storage stage, thereby further reducing the total data volume. Traditional compression methods all use similar algorithms, and cannot achieve further compression effects when used in series.
[0048] In addition, there are related studies that use compressed sampling in a specific scenario, such as image compression. Since the data features of this type of scenario are fixed (such as fixed to images), the sparse transform domain of the data is fixed, and existing studies have used fixed transform domain compressed sampling schemes. However, in the time series database scenario, time series data may be of different types, including equipment performance data, alarm data, and business data. The data may be sparse in one domain and not sparse in another domain. Therefore, the existing compressed sampling method processes data according to the fixed transform domain, which will result in a loss in compression rate or restoration accuracy.
[0049] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.
[0050] According to an embodiment of the present application, a method embodiment of a data compression method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0051] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data compression method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0052] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0053] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data compression method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned data compression method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0054] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0055] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).
[0056] In the above operating environment, the embodiment of the present application provides a data compression method, such as Figure 2 As shown, the method comprises the following steps:
[0057] Step S202, determining a first sparsity index of the time series data to be compressed;
[0058] In the technical solution provided in step S202, after determining the ratio of non-zero elements in the data to be compressed to all elements in the data to be compressed, the data compression method also includes: when the first sparsity index is less than the first preset threshold, determining that the domain where the data to be compressed is located is the target transformation domain, wherein the data to be compressed includes time series data, and the domain where the data to be compressed includes the time domain. The above-mentioned first sparsity index includes the ratio of non-zero elements to all elements in the data to be compressed. It should be noted that the non-zero elements and non-zero coefficients in this application refer to significant non-zero elements and significant non-zero coefficients. That is to say, elements whose values are within the preset range will not be considered as non-zero elements or non-zero coefficients.
[0059] In some embodiments of the present application, time series data of the operation of all professional equipment in the network domain can be continuously collected from cloud network devices, and sparsity analysis can be performed on the time series data to calculate the proportion of non-zero elements or other sparsity measurement indicators of the time series data.
[0060] Step S204, when the first sparsity index is not less than the first preset index threshold, determining the transform coefficients corresponding to the to-be-compressed data in multiple transform domains;
[0061] In the technical solution provided in step S204, the transform domain includes at least one of the following: a discrete cosine transform domain, a discrete wavelet transform domain, and a discrete Fourier transform domain.
[0062] As an optional implementation, determining the transform coefficients corresponding to the data to be compressed in each transform domain includes: performing discrete cosine transform on the compressed data to obtain discrete cosine transform coefficients corresponding to the discrete cosine transform domain; performing discrete wavelet transform on the compressed data to obtain discrete wavelet transform coefficients corresponding to the discrete wavelet transform domain; performing discrete Fourier transform on the compressed data to obtain discrete Fourier transform coefficients corresponding to the discrete Fourier transform domain. The specific calculation formulas for various transform coefficients are as follows:
[0063] X w =DWT(x)
[0064] X c =DCT(x)
[0065] X f =DFT(x)
[0066] In the above formula, x is the original time series data, X w is the coefficient after DWT transformation, X c is the coefficient after DCT transformation, X f are the coefficients after DFT transformation.
[0067] In some embodiments of the present application, the corresponding transformation method can be selected according to the specific characteristics of different data. For example, DWT transformation can be used preferentially for stationary signals, while DCT or DFT transformation can be used preferentially for periodic signals. In this way, in scenarios where computing resources are limited or time limits are high, when it is impossible to perform various transformations on all the data to be processed to determine the optimal transformation domain, the transformation domain in which the data to be processed may have sparsity can be quickly determined.
[0068] Step S206, determining a second sparsity index and a uniform distribution index corresponding to each of the multiple transform domains according to the transform coefficients corresponding to the multiple transform domains, and determining a target transform domain in each of the transform domains according to the second sparsity index and the uniform distribution index;
[0069] In the technical solution provided in step S206, determining the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index includes: when the second sparsity index corresponding to each transform domain is not less than the second preset index threshold, determining the transform domain with the largest corresponding uniform distribution index as the target transform domain; when there is a non-zero coefficient ratio less than the second preset index threshold in the second sparsity index corresponding to each transform domain, determining the transform domain with the corresponding non-zero coefficient ratio less than the first preset coefficient threshold as the target transform domain; when there are multiple non-zero coefficient ratios less than the second preset threshold in the second sparsity index corresponding to each transform domain, determining the transform domain with the smallest corresponding non-zero coefficient ratio as the target transform domain.
[0070] In some embodiments of the present application, the first sparsity index includes the ratio of non-zero elements in the data to be compressed, the second sparsity index corresponding to each transform domain includes the ratio of non-zero coefficients in the transform coefficients corresponding to each transform domain, and the uniform distribution index corresponding to each transform domain includes the Gini index. Specifically, the ratio of non-zero coefficients in the coefficients obtained by DWT, DCT and DFT transformation is set to R w , R c and R f , then the calculation formula for the proportion of each non-zero coefficient is as follows:
[0071]
[0072]
[0073]
[0074] In the above formula, ∥X w ∥0, ∥X c ∥0 and ∥X f ∥0 means X w , X c and X f is the number of non-zero elements in , and N is the total number of transform coefficients.
[0075] Set the Gini index of DWT, DCT and DFT transform coefficients to G w , G c and G wf , then the calculation formula of Gini index is as follows:
[0076] G w =Gini(|X w |)
[0077] G c =Gini(|X c |)
[0078] G f =Gini(|X f |)
[0079] In the above formula, |X w |、|X c | and |X f | respectively represent X w , X c and X f The absolute value of .
[0080] In some embodiments of the present application, R w , R c and R f , if the three are significantly different, then choose the transformation with a smaller proportion of non-zero coefficients. If the three are not very different, then further compare G w , G ct and G f Select the transformation with the larger Gini index.
[0081] As an optional implementation, a norm threshold may be used to determine the sparsity of transformed (DWT, DCT or DFT) data to screen out sparse coefficients.
[0082] ∥x′∥0<θ
[0083] Among them, x′ is the transformed coefficient, θ is the sparsity threshold, and ∥x′∥0 represents the number of non-zero elements. If the number of non-zero elements is less than θ, it can be considered that most of the transformed coefficients are concentrated in a few values, which indicates that the original data has good compression potential in the transformed domain and is suitable for compressed sampling.
[0084] It should be noted that the reason why it is necessary to determine the target transform domain and convert the data to the target transform domain before performing compressed sampling is that data with different characteristics will have different coefficient distribution characteristics in different transform domains. As a result, if all data are converted to the same transform domain, some data will have better coefficient sparsity in the transform domain, which is suitable for compressed sampling, while some data will have poor coefficient sparsity and is not suitable for compressed sampling.
[0085] Specifically, Figure 3 As shown in the figure, when the data is suitable for time domain processing instead of DCT domain processing, the reconstructed data after time domain processing has good sparsity, and the reconstructed data after time domain processing basically overlaps with the original data. The reconstructed data after DCT domain processing is closer to uniform distribution. It can be seen that the data is more suitable for compression processing in the time domain rather than the DCT domain.
[0086] and Figure 4The comparison between the reconstructed data and the original data after DCT processing is shown in the figure. Figure 4 The reconstructed data and the original data basically overlap, indicating that the data is suitable for compressed sampling in the DCT domain. Figure 5 The figure shows the comparison between the reconstructed data and the original data after DWT processing for the compressed data which is not sparse in the time domain and is not suitable for DWT processing. It can be seen that there is little overlap between the reconstructed data and the original data, indicating that the data is not suitable for compression sampling in the DWT domain. Figure 3 , Figure 4 and Figure 5 It can be seen that only when different data are compressed and sampled in an adapted transform domain can good compression sampling and reconstruction effects be achieved.
[0087] Specifically, Figure 3 The data in is a time series data with obvious periodic and stationary characteristics. By comparison, it can be seen that directly sampling and reconstructing the data in the time domain can better preserve its original waveform characteristics and achieve a higher reconstruction accuracy. Although transforming it to the discrete cosine transform (DCT) domain and then sampling and reconstructing it can utilize its frequency domain sparse characteristics to a certain extent, the reconstruction effect is reduced. This shows that for such stationary periodic signals, the time domain processing method is more advantageous.
[0088] In contrast, Figure 4 and Figure 5 The experimental data shown contains more non-stationary and mutation components. It can be seen that after transforming the data into the DCT domain, the signal energy can be better concentrated on a few low-frequency components, and the compact representation effect of the frequency domain can be used for sampling and reconstruction, thereby obtaining good reconstruction quality. On the contrary, if the data is processed in the time domain or wavelet transform (DWT) domain, its sparse structure cannot be fully exploited, resulting in a significant decrease in reconstruction accuracy.
[0089] Figure 6 The comparison results between the reconstructed data and the original data of the compressed data which are not sparse in the time domain after time domain processing are shown in Fig. Figure 7 The comparison between the reconstructed data and the original data after DWT processing is shown in Figure 1. Figure 6 and Figure 7 It can be seen that the transform domain selection has an important impact on the performance of compressed sensing. Figure 6 The waveform of the data in the converted image is quite different from the original data and is not sparse. Figure 7 The reconstructed data and the original data in the image are basically overlapped.
[0090] comprehensive Figure 3-Figure 7 It can be seen that the transform domain selected when compressing data will have a great influence on the restoration accuracy of the reconstructed data.
[0091] Step S208: compress and sample the data to be compressed according to the target transform domain to obtain compressed data corresponding to the data to be compressed.
[0092] In the technical solution provided in step S208, the step of compressing and sampling the data to be compressed according to the target transform domain includes: converting the data to be compressed into the target transform domain; and compressing and sampling the data to be compressed converted into the target transform domain using an observation matrix.
[0093] In some embodiments of the present application, Figure 8 As shown, when compressing the data, the signal x is not sampled directly, but through a linear measurement process, which can be expressed as an observation matrix To describe, represents the field of real numbers, and Figure 8 R in is also a real number domain, where M<<N. The measurement process can be expressed as:
[0094] y=Φx=ΦΨθ
[0095] Where y is the M-dimensional measurement vector, representing the projection of the original signal x under the observation matrix Φ, Ψ is the basis matrix, such as the discrete Fourier transform, discrete wavelet transform or discrete cosine transform basis, θ and Figure 8 The W in is a sparse coefficient vector.
[0096] from Figure 8 It can be seen that in this application, compression sampling is performed after sparse decomposition of the data, thereby ensuring the compression effect and reducing the loss of compression rate.
[0097] The compressed data can then be stored in a database for transmission or further processing.
[0098] In some embodiments of the present application, Fig. 9 As shown, the complete process of the data compression method provided in the embodiment of the present application includes the following steps:
[0099] Step S902, after collecting the time series data, perform sparsity analysis on the time series database data in the time domain;
[0100] Step S904, judging whether the data is significantly sparse in the time domain, if yes, jump to step S914, otherwise jump to step S906;
[0101] Step S906, performing DWT, DCT and DFT transformations on the data respectively;
[0102] Step S908, calculating the non-zero coefficient ratios of DWT, DCT and DFT transform coefficients;
[0103] Step S910, determine whether the proportion of non-zero coefficients in various types of transform coefficients is significantly different. If so, select the transform with a smaller proportion of non-zero coefficients as the selected transform domain. Otherwise, calculate the Gini index of DWT, DCT and DFT, and determine the transform domain with a larger Gini index as the selected transform domain.
[0104] Step S912, perform sparsity determination on the transformed data (determine the sparsity of the data by using the norm threshold), if the determination result is sparse, jump to step S914, otherwise end the compression process;
[0105] Step S914, compressing and sampling the transformed data, including using a Gaussian matrix or other observation matrix to perform compressive sampling in the selected change domain;
[0106] Step S916: store the compressed sampling data in a database.
[0107] As an optional implementation, the process of retrieving and restoring compressed data includes the following steps: retrieving target compressed data corresponding to the retrieval instruction according to the retrieval instruction; determining basis vectors and sparse coefficient vectors of the target compressed data; and decompressing the target compressed data based on the basis vectors and the sparse coefficient vectors.
[0108] As an optional implementation, when reconstructing the compressed data, an orthogonal matching pursuit algorithm (OMP) can be used to gradually construct a sparse representation by iteratively selecting atoms that are most relevant to the residual. Assume that the signal is sparse or can be transformed into a sparse representation through an appropriate transformation Ψ, namely:
[0109] x=Ψθ
[0110] where θ is a sparse coefficient vector and Ψ is a basis matrix such as a discrete Fourier transform, discrete wavelet transform, or discrete cosine transform basis.
[0111] In some embodiments of the present application, there is also provided a Fig.10 The decompression process shown includes the following steps:
[0112] Step S1002, retrieving compressed data to be restored;
[0113] Step S1004: reconstruct the compressed data using an orthogonal matching pursuit algorithm.
[0114] In summary, by adopting a first sparsity index for determining the time series data to be compressed; determining the transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than a first preset index threshold; determining the second sparsity index and uniform distribution index corresponding to each transform domain in the multiple transform domains according to the transform coefficients corresponding to the multiple transform domains, and determining the target transform domain in each transform domain according to the second sparsity index and the uniform distribution index; compressing and sampling the data to be compressed according to the target transform domain to obtain compressed data corresponding to the data to be compressed, determining the target compression domain by determining the sparsity index and uniform distribution index of the data to be compressed, and converting the data to be compressed to the target compression domain for compression, the purpose of ensuring the sparsity of the data after conversion is achieved, thereby achieving the technical effect of reducing the loss of compression accuracy and restoration rate caused by the compression process, and further solving the technical problem of the compression rate and restoration accuracy being damaged due to converting data to a fixed transform domain and then compressing and sampling in the related technology.
[0115] In addition, the data compression method provided in the embodiment of the present application also utilizes the sparse characteristics of time series data in the transform domain, directly performs compression sampling, reduces the number of sampled data points, and compresses in different dimensions compared to traditional methods (such as ZIP), thus developing a new compression dimension. A method for adaptive transform domain selection is also provided, which can adaptively select the optimal transform basis according to the characteristics of different time series data in actual network operation data, improve the sparsity of the transform domain, and thus improve compression efficiency. However, the compressed sensing method in the related art usually adopts a fixed transform domain, which cannot adapt to the diversity of data features in actual network operation data.
[0116] Furthermore, the compression method provided in the embodiment of the present application can also be used in tandem with traditional compression algorithms such as ZIP to solve problems at different levels to achieve better compression effects.
[0117] It should be noted that the data compression method provided in the embodiment of the present application is particularly suitable for compressing and sampling the operation data of the operator network, and fully considers the non-fixed nature of data characteristics, and has a wider applicability.
[0118] The present application embodiment provides a data compression system, Fig.11 This is a schematic diagram of the system structure. Fig.11As can be seen in FIG, the system includes a data compression module 110, a data storage module 112, and a data reconstruction module 114. The data compression module is used to perform the following data compression method: determining a first sparsity index of the time series data to be compressed; determining the transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than a first preset index threshold; determining a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains according to the transform coefficients corresponding to the multiple transform domains, and determining a target transform domain in each transform domain according to the second sparsity index and the uniform distribution index; compressing and sampling the data to be compressed according to the target transform domain to obtain compressed data corresponding to the data to be compressed.
[0119] The data storage module 112 is used to store the compressed data in the database and further perform lossless compression on the data. After receiving the search instruction, it determines the data to be decompressed and performs lossless decompression, and then retrieves the data and sends it to the data reconstruction module 114 to reconstruct the data using an orthogonal tracking algorithm.
[0120] The present application provides a data compression device. Fig.12 is a schematic diagram of the structure of the device. Fig.12 It can be seen that the device includes: a first processing module 120, which is used to determine the first sparsity index of the time series data to be compressed; a second processing module 122, which is used to determine the transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than the first preset index threshold; a third processing module 124, which is used to determine the second sparsity index and uniform distribution index corresponding to each transform domain in the multiple transform domains based on the transform coefficients corresponding to the multiple transform domains, and determine the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index; a fourth processing module 126, which is used to determine the second sparsity index and uniform distribution index corresponding to each transform domain in the multiple transform domains based on the transform coefficients corresponding to the multiple transform domains, and determine the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index.
[0121] In some embodiments of the present application, after determining the ratio of non-zero elements in the data to be compressed to all elements in the data to be compressed, the first processing module 120 is also used to: when the first sparsity index is less than a first preset threshold, determine that the domain where the data to be compressed is located is the target transformation domain, wherein the data to be compressed includes time series data, and the domain where the data to be compressed is located includes the time domain.
[0122] In some embodiments of the present application, the transform domain includes at least one of the following: a discrete cosine transform domain, a discrete wavelet transform domain, and a discrete Fourier transform domain.
[0123] In some embodiments of the present application, the steps in which the second processing module 122 determines the transform coefficients corresponding to the data to be compressed in each transform domain include: performing a discrete cosine transform on the compressed data to obtain discrete cosine transform coefficients corresponding to the discrete cosine transform domain; performing a discrete wavelet transform on the compressed data to obtain discrete wavelet transform coefficients corresponding to the discrete wavelet transform domain; performing a discrete Fourier transform on the compressed data to obtain discrete Fourier transform coefficients corresponding to the discrete Fourier transform domain.
[0124] In some embodiments of the present application, the step in which the third processing module 124 determines the target transform domain in each transform domain based on the second sparsity index and the uniform distribution index includes: when the second sparsity index corresponding to each transform domain is not less than the second preset index threshold, determining the transform domain with the largest corresponding uniform distribution index as the target transform domain; when there is a non-zero coefficient ratio less than the second preset index threshold in the second sparsity index corresponding to each transform domain, determining the transform domain with the corresponding non-zero coefficient ratio less than the first preset coefficient threshold as the target transform domain; when there are multiple non-zero coefficient ratios less than the second preset threshold in the second sparsity index corresponding to each transform domain, determining the transform domain with the smallest corresponding non-zero coefficient ratio as the target transform domain.
[0125] In some embodiments of the present application, the first sparsity indicator includes the ratio of non-zero elements in the data to be compressed, the second sparsity indicator corresponding to each transform domain includes the ratio of non-zero coefficients in the transform coefficients corresponding to each transform domain, and the uniform distribution indicator corresponding to each transform domain includes the Gini index.
[0126] In some embodiments of the present application, the step of the fourth processing module 126 compressing and sampling the data to be compressed according to the target transform domain includes: converting the data to be compressed into the target transform domain; and compressing and sampling the data to be compressed converted into the target transform domain using an observation matrix.
[0127] In some embodiments of the present application, the data compression device is also used to: retrieve target compressed data corresponding to the retrieval instruction based on the retrieval instruction; determine the basis vector and the sparse coefficient vector of the target compressed data; and decompress the target compressed data based on the basis vector and the sparse coefficient vector.
[0128] It should be noted that the various modules in the above-mentioned data compression device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0129] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following data compression method: determine a first sparsity index of the time series data to be compressed; when the first sparsity index is not less than a first preset index threshold, determine the transform coefficients corresponding to the data to be compressed in multiple transform domains; determine a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains based on the transform coefficients corresponding to the multiple transform domains, and determine a target transform domain in each transform domain based on the second sparsity index and the uniform distribution index; compress and sample the data to be compressed based on the target transform domain to obtain compressed data corresponding to the data to be compressed.
[0130] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory and a processor, the processor being used to run a program stored in the memory, wherein the following data compression method is executed when the program is running: determining a first sparsity index of time series data to be compressed; determining transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than a first preset index threshold; determining a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains based on the transform coefficients corresponding to the multiple transform domains, and determining a target transform domain in each transform domain based on the second sparsity index and the uniform distribution index; compressing and sampling the data to be compressed based on the target transform domain to obtain compressed data corresponding to the data to be compressed.
[0131] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which executes the following data compression method when executed by a processor: determining a first sparsity index of time series data to be compressed; determining transform coefficients corresponding to the data to be compressed in multiple transform domains when the first sparsity index is not less than a first preset index threshold; determining a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains based on the transform coefficients corresponding to the multiple transform domains, and determining a target transform domain in each transform domain based on the second sparsity index and the uniform distribution index; compressing and sampling the data to be compressed based on the target transform domain to obtain compressed data corresponding to the data to be compressed.
[0132] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0133] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0137] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A data compression method, characterized in that: include: Determining a first sparsity index of the time series data to be compressed; When the first sparsity index is not less than a first preset index threshold, determining transform coefficients corresponding to the to-be-compressed data in a plurality of transform domains; Determine a second sparsity index and a uniform distribution index corresponding to each transform domain in the multiple transform domains according to the transform coefficients corresponding to the multiple transform domains, and determine a target transform domain in each transform domain according to the second sparsity index and the uniform distribution index; The data to be compressed is compressed and sampled according to the target transform domain to obtain compressed data corresponding to the data to be compressed.
2. The data compression method according to claim 1, characterized in that: Determining a target transform domain in each transform domain according to the second sparsity indicator and the uniform distribution indicator includes: When the second sparsity index corresponding to each transform domain is not less than a second preset index threshold, determining the transform domain with the largest corresponding uniform distribution index as the target transform domain; In the case where there is a non-zero coefficient ratio less than the second preset indicator threshold value in the second sparsity index corresponding to each transform domain, determining the transform domain whose corresponding non-zero coefficient ratio is less than the second preset indicator threshold value as the target transform domain; When there are multiple non-zero coefficient ratios less than the second preset indicator threshold in the second sparsity indicators corresponding to the respective transform domains, the transform domain corresponding to the smallest non-zero coefficient ratio is determined as the target transform domain.
3. The data compression method according to claim 1, characterized in that: The first sparsity index includes the ratio of non-zero elements in the data to be compressed, the second sparsity index corresponding to each transform domain includes the ratio of non-zero coefficients in the transform coefficients corresponding to each transform domain, and the uniform distribution index corresponding to each transform domain includes the Gini index.
4. The data compression method according to claim 1, characterized in that: After determining the ratio of the non-zero elements in the data to be compressed to all the elements in the data to be compressed, the data compression method further includes: When the first sparsity index is less than the first preset index threshold, the domain where the data to be compressed is located is determined to be the target transformation domain, wherein the data to be compressed includes time series data, and the domain where the data to be compressed is located includes a time domain.
5. The data compression method according to claim 1, characterized in that: Compressing and sampling the data to be compressed according to the target transform domain includes: Converting the data to be compressed into the target transform domain; The observation matrix is used to perform compression sampling on the data to be compressed that is converted into the target transform domain.
6. The data compression method according to claim 1, characterized in that: The transform domain includes at least one of the following: a discrete cosine transform domain, a discrete wavelet transform domain, and a discrete Fourier transform domain.
7. The data compression method according to claim 6, characterized in that: Determining the transform coefficients corresponding to the to-be-compressed data in each transform domain includes: Performing discrete cosine transform on the data to be compressed to obtain discrete cosine transform coefficients corresponding to the discrete cosine transform domain; Performing discrete wavelet transform on the data to be compressed to obtain discrete wavelet transform coefficients corresponding to the discrete wavelet transform domain; Performing a discrete Fourier transform on the data to be compressed to obtain discrete Fourier transform coefficients corresponding to the discrete Fourier transform domain.
8. The data compression method according to claim 1, characterized in that: The data compression method further comprises: Retrieving target compressed data corresponding to the retrieval instruction according to the retrieval instruction; Determining basis vectors and a sparse coefficient vector of the target compressed data; The target compressed data is decompressed according to the basis vectors and the sparse coefficient vectors.
9. A data compression device, characterized in that: include: A first processing module, used to determine a first sparsity index of the time series data to be compressed; A second processing module, configured to determine transform coefficients corresponding to the to-be-compressed data in a plurality of transform domains when the first sparsity index is not less than a first preset index threshold; a third processing module, configured to determine a second sparsity index and a uniform distribution index corresponding to each of the plurality of transform domains according to the transform coefficients corresponding to the plurality of transform domains, and determine a target transform domain in each of the transform domains according to the second sparsity index and the uniform distribution index; The fourth processing module is used to determine the second sparsity index and the uniform distribution index corresponding to each transform domain in the multiple transform domains according to the transform coefficients corresponding to the multiple transform domains, and determine the target transform domain in each transform domain according to the second sparsity index and the uniform distribution index.
10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the data compression method according to any one of claims 1 to 8.
11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the data compression method according to any one of claims 1 to 8 when running.
12. A computer program product, characterized in that The invention comprises a computer program, which implements the data compression method according to any one of claims 1 to 8 when being executed by a processor.
Citation Information
Cited By
Method for the energy-efficient operation of a wireless monitoring sensor and a correspondingly configured monitoring sensor
DE102024137700A1
