Data compression method and device based on abnormal separation, electronic equipment and storage medium

Abstract

The application provides a data compression method and device based on abnormal separation, electronic equipment and storage medium, and relates to the technical field of data compression. The method comprises the following steps: according to normal values and abnormal values in a to-be-compressed data sequence, the to-be-compressed data sequence is divided into at least two data sequences, one data sequence in the at least two data sequences contains the normal values, and other data sequences contain the abnormal values; and data compression is respectively performed on the at least two data sequences. According to the method, the normal values and the abnormal values in the to-be-compressed data sequence are separated and then subjected to data compression, so that the time and space costs of data compression can be reduced, and the data compression efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and in particular to a data compression method, apparatus, electronic device, and storage medium based on anomaly separation. Background Technology

[0002] The Internet of Things (IoT) and time series analytics are two rapidly developing fields that play crucial roles in numerous applications. With the proliferation of IoT devices, the types and quantities of time series data have exploded, making efficient compression and storage of this data increasingly important. Bit stuffing is a classic component of traditional compression algorithms, used to store values ​​within data blocks using the same bit width and removing leading zeros to achieve compression.

[0003] However, during the process of filling the sequence bits, if some extreme outliers occur, the bit width of the entire sequence will increase excessively, leading to an increase in the time and space costs of data compression. Summary of the Invention

[0004] This invention provides a data compression method, apparatus, electronic device, and storage medium based on anomaly separation, to address the shortcomings of existing technologies where outliers increase the time and space costs of data compression. The method divides the data sequence to be compressed into at least two data sequences based on normal and outlier values, and then compresses each of the at least two data sequences separately. This achieves the effect of separating and compressing outliers, reducing the time and space costs of data compression, and improving data compression efficiency.

[0005] This invention provides a data compression method based on anomaly separation, comprising the following steps:

[0006] Based on the normal values ​​and outliers in the data sequence to be compressed, the data sequence to be compressed is divided into at least two data sequences. One of the at least two data sequences contains the normal values, and the other data sequences contain the outliers. The outliers are used to indicate one or more values ​​whose values ​​deviate significantly from the values ​​of most of the values ​​in the data sequence to be compressed. The number of normal values ​​is greater than the number of outliers.

[0007] Data compression is performed on the at least two data sequences respectively.

[0008] According to the present invention, a data compression method based on anomaly separation is provided, wherein dividing the data sequence to be compressed into at least two data sequences according to normal values ​​and anomalies in the data sequence to be compressed includes:

[0009] The first boundary value and the second boundary value are determined based on the target features of each value in the data sequence to be compressed. The target features include the value size and / or the bit width. The first boundary value is smaller than the second boundary value. The bit width is used to indicate the number of bits required to store the binary value of the value.

[0010] The data sequence to be compressed is divided into an upper outlier sequence, a normal value sequence, and a lower outlier sequence based on the first boundary value and the second boundary value. The upper outlier sequence includes values ​​in the data sequence to be compressed that are less than the first boundary value. The lower outlier sequence includes values ​​in the data sequence to be compressed that are greater than the second boundary value. The normal value sequence includes values ​​in the data sequence to be compressed that are greater than or equal to the first boundary value and less than or equal to the second boundary value.

[0011] According to the present invention, a data compression method based on anomaly separation is provided, wherein the target feature includes value magnitude, and the step of determining the first boundary value and the second boundary value for dividing the data sequence to be compressed based on the target feature of each value in the data sequence to be compressed includes:

[0012] The values ​​in the data sequence to be compressed are sorted according to their size to obtain a sorted sequence;

[0013] In the sorted sequence, different values ​​are selected as the first boundary value and the second boundary value to obtain multiple sets of boundary value combinations. The cost corresponding to each set of boundary value combinations is calculated. The cost is used to characterize the compressed storage cost.

[0014] The values ​​in the boundary value combination that minimizes the corresponding cost are determined as the first boundary value and the second boundary value.

[0015] According to the present invention, a data compression method based on anomaly separation is provided, wherein the step of traversing and selecting different values ​​as the first boundary value and the second boundary value in the sorted sequence to obtain multiple sets of boundary value combinations, and calculating the cost corresponding to each set of boundary value combinations, includes:

[0016] The values ​​in the sorted sequence are determined as the first boundary values ​​in ascending order of their values.

[0017] After determining the first boundary value, the values ​​in the sorted sequence are then determined as the second boundary values ​​in order of ascending value and value greater than the first boundary value, and the corresponding costs are calculated to obtain the costs corresponding to each combination of boundary values.

[0018] According to the present invention, a data compression method based on anomaly separation is provided, wherein the target feature includes bit width, and the step of determining the first boundary value and the second boundary value for dividing the data sequence to be compressed based on the target feature of each value in the data sequence to be compressed includes:

[0019] Determine the bit width of each value in the data sequence to be compressed;

[0020] The first boundary value and the second boundary value are determined based on the bit width distribution corresponding to the bit width of each value.

[0021] According to the present invention, a data compression method based on anomaly separation, wherein determining the first boundary value and the second boundary value according to the bit width distribution corresponding to the bit width of each value includes:

[0022] Iterate through each bit width within the first bit width range, then iterate through and select all values ​​in the data sequence to be compressed as normal values, determine the target bit width and starting value corresponding to the normal value, the target bit width is used to indicate the optimal bit width representing the normal value in the data sequence to be compressed, and the starting value is used to indicate the optimal starting value representing the normal value in the data sequence to be compressed;

[0023] Based on the target bit width and the starting value, all values ​​in the data sequence to be compressed are selected as the starting value of the normal value. Then, outliers are selected and the corresponding costs are calculated. The costs are used to characterize the compression storage cost.

[0024] The values ​​that minimize the corresponding costs are determined as the first boundary value and the second boundary value.

[0025] According to the present invention, a data compression method based on anomaly separation is provided, wherein the target feature includes value size and bit width, and the step of determining the first boundary value and the second boundary value for dividing the data sequence to be compressed based on the target feature of each value in the data sequence to be compressed includes:

[0026] Obtain the median of the data sequence to be compressed;

[0027] The normal values ​​are selected by iterating through the data sequence to be compressed, with the median as the center.

[0028] Determine the bit width range of the normal value, and determine the first boundary value and the second boundary value based on the bit width range of the normal value.

[0029] According to a data compression method based on anomaly separation provided by the present invention, obtaining the median of the data sequence to be compressed includes:

[0030] Perform differential processing on each value in the data sequence to be compressed to obtain the processed data sequence;

[0031] Observe the normal distribution of each value in the processed data sequence;

[0032] The median is determined based on the normal distribution of the values.

[0033] According to the present invention, a data compression method based on anomaly separation is provided, wherein the at least two data sequences include a target data sequence, the target data sequence includes the normal value, and the data compression of the at least two data sequences includes:

[0034] The minimum bit width required to store the normal values ​​is determined based on the number of normal values.

[0035] The normal value is stored using the minimum bit width, where the first value stored using the minimum bit width is the starting value of the normal value, and the last value stored using the minimum bit width is the ending value of the normal value.

[0036] The present invention also provides a data compression device based on anomaly separation, comprising the following modules:

[0037] The data partitioning module is used to divide the data sequence to be compressed into at least two data sequences based on normal values ​​and outliers in the data sequence to be compressed. One of the data sequences contains the normal values, and the other data sequences contain the outliers. The outliers are used to indicate one or more values ​​whose values ​​deviate significantly from the values ​​of most values ​​in the data sequence to be compressed. The number of normal values ​​is greater than the number of outliers.

[0038] A data compression module is used to compress the at least two data sequences respectively.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the data compression method based on anomaly separation as described above.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data compression method based on anomaly separation as described above.

[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data compression method based on anomaly separation as described above.

[0042] The present invention provides a data compression method, apparatus, electronic device, and storage medium based on anomaly separation. The method divides the data sequence to be compressed into at least two data sequences based on normal and outlier values. One of the at least two data sequences contains the normal values, while the other data sequences contain the outlier values. Data compression is then performed on each of the at least two data sequences. This method separates the normal and outlier values ​​in the data sequence and performs data compression accordingly, thereby reducing the time and space costs of data compression and improving data compression efficiency. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the data compression method based on anomaly separation provided by the present invention.

[0045] Figure 2 This is a flowchart illustrating the method for dividing a data sequence to be compressed into at least two data sequences provided by the present invention.

[0046] Figure 3 This is an example of how the present invention separates and classifies outliers based on precise value magnitude.

[0047] Figure 4 This is a schematic diagram of the sequence sorting provided by the present invention.

[0048] Figure 5 This is a schematic diagram of a scenario for classifying outliers provided by the present invention.

[0049] Figure 6 This is a schematic diagram of the cumulative counting provided by the present invention.

[0050] Figure 7 This is a flowchart illustrating the method for compressing at least two data sequences provided by the present invention.

[0051] Figure 8 This is a schematic diagram of the data compression device based on anomaly separation provided by the present invention.

[0052] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] The Internet of Things (IoT) and time series analytics are two rapidly developing fields that play crucial roles in numerous applications. With the proliferation of IoT devices, the types and quantities of time series data have exploded, making efficient compression and storage of this data increasingly important. Bit stuffing is a classic component of traditional compression algorithms, used to store values ​​within data blocks using the same bit width and removing leading zeros to achieve compression.

[0055] However, during the process of filling the sequence bits, if some extreme outliers occur, the bit width of the entire sequence will increase excessively, leading to an increase in the time and space costs of data compression.

[0056] In view of this, embodiments of the present invention provide a data compression method based on anomaly separation. The method divides the data sequence to be compressed into at least two data sequences according to the normal values ​​and outliers in the sequence. One of the at least two data sequences contains the normal values, and the other data sequences contain the outliers. Data compression is then performed on each of the at least two data sequences. This method separates the normal and outliers in the data sequence and performs data compression separately, which can reduce the time and space costs of data compression and improve data compression efficiency.

[0057] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0058] Figure 1 This is a flowchart illustrating the anomaly-based data compression method provided by the present invention. The anomaly-based data compression method can be applied to electronic devices, which can be various types of devices with information processing capabilities. For example, the electronic device may include a personal computer, laptop, handheld computer, or server; the electronic device may also be a mobile terminal, such as a mobile phone, in-vehicle computer, tablet computer, or projector. Figure 1 As shown, the method may include the following steps 101 to 102:

[0059] Step 101: Based on the normal values ​​and outliers in the data sequence to be compressed, divide the data sequence to be compressed into at least two data sequences. One of the at least two data sequences contains the normal values, and the other data sequences contain the outliers. The outliers are used to indicate one or more values ​​whose values ​​deviate significantly from the values ​​of most of the values ​​in the data sequence to be compressed. The number of normal values ​​is greater than the number of outliers.

[0060] It should be noted that identifying normal and outlier values ​​in the data sequence to be compressed, and dividing the data sequence into at least two data sequences—one for normal values ​​and one for outliers—lays the foundation for subsequent compression. The number of identified normal values ​​is greater than the number of outliers, which reduces storage costs. Outliers are one or more values ​​whose magnitudes significantly deviate from the magnitudes of most values ​​in the data sequence to be compressed.

[0061] There are many ways to divide the data sequence to be compressed into at least two data sequences based on the normal and abnormal values ​​in the data sequence to be compressed. For example, the normal and abnormal values ​​in the data sequence to be compressed can be determined based on a preset range of normal values ​​or by statistically determining the range of normal values, and then the data sequence to be compressed can be divided. This invention does not limit the method of dividing the data sequence to be compressed into at least two data sequences based on the normal and abnormal values ​​in the data sequence to be compressed.

[0062] Step 102: Compress the data for each of the at least two data sequences.

[0063] It should be noted that there are many ways to compress the at least two data sequences separately. For example, it can be to determine the required bit width for each value in each data sequence and store it with the corresponding bit width, or it can be to store it according to a preset bit width, etc. The present invention does not limit the method of compressing the at least two data sequences separately.

[0064] It is understood that the method provided by this invention, which separates normal values ​​and outliers and stores them separately, namely, improves the bit stuffing algorithm based on outlier separation for data processing, and realizes a more efficient data compression method.

[0065] In some embodiments, dividing the data sequence to be compressed into at least two data sequences can be divided into three data sequences, that is, determining two boundary values ​​and dividing the data sequence to be compressed into an upper outlier sequence, a normal value sequence, and a lower outlier sequence.

[0066] Figure 2This is a flowchart illustrating the method for dividing a data sequence to be compressed into at least two data sequences provided by the present invention. Figure 2 As shown, dividing the data sequence to be compressed into at least two data sequences based on normal and outlier values ​​in the data sequence to be compressed may include:

[0067] Step 201: Determine the first boundary value and the second boundary value for dividing the data sequence to be compressed based on the target features of each value in the data sequence to be compressed. The target features include the value size and / or the bit width. The first boundary value is smaller than the second boundary value. The bit width is used to indicate the number of bits required to store the binary value of the value.

[0068] It should be noted that determining the first and second boundary values ​​for dividing the data sequence to be compressed based on the target characteristics of each value in the data sequence to be compressed specifies that the implementation scheme for anomaly separation is not a simple heuristic, but rather the first and second boundary values ​​are determined based on the corresponding values ​​and / or location storage costs.

[0069] Step 202: Divide the data sequence to be compressed into an upper outlier sequence, a normal value sequence, and a lower outlier sequence according to the first boundary value and the second boundary value. The upper outlier sequence includes values ​​in the data sequence to be compressed that are less than the first boundary value. The lower outlier sequence includes values ​​in the data sequence to be compressed that are greater than the second boundary value. The normal value sequence includes values ​​in the data sequence to be compressed that are greater than or equal to the first boundary value and less than or equal to the second boundary value.

[0070] It should be noted that after determining the first and second boundary values, the data sequence to be compressed can be divided into upper outlier sequence, normal value sequence and lower outlier sequence, so as to compress and store them separately.

[0071] Understandably, after dividing the data sequence to be compressed into upper outlier sequences, normal value sequences, and lower outlier sequences, the outlier sequences require additional storage of their positional information during subsequent storage, incurring a higher cost than the normal value sequences. The normal value sequences, however, are stored in a more convenient, direct, and compact manner. Therefore, determining the first and second boundary values ​​for dividing the data sequence to be compressed based on the magnitude and / or bit width of each value in the sequence can improve the accuracy of sequence division and reduce the overall compression and storage cost of the data sequence.

[0072] In some embodiments, the method for determining the first boundary value and the second boundary value of the data sequence to be compressed based on the target characteristics of each value in the data sequence to be compressed includes three types: dividing according to value size, dividing according to bit width, and dividing according to both value size and bit width. The division method mainly involves a two-dimensional search, that is, finding two boundary values, that is, selecting different methods to perform traversal search, which will be described in detail below.

[0073] Method 1: Separate outliers in the sequence based on the precise value size.

[0074] In an embodiment of the present invention, the target feature includes value size. Determining the first boundary value and the second boundary value of the data sequence to be compressed based on the target feature of each value in the data sequence to be compressed may include: sorting the values ​​in the data sequence to be compressed according to their value size to obtain a sorted sequence; traversing and selecting different values ​​as the first boundary value and the second boundary value in the sorted sequence to obtain multiple sets of boundary value combinations; calculating the cost corresponding to each set of boundary value combinations, wherein the cost is used to characterize the compression storage cost; and determining the value in the boundary value combination with the smallest corresponding cost as the first boundary value and the second boundary value.

[0075] It should be noted that separating outliers based on precise values ​​to divide the outlier sequence is the most accurate method. This approach considers all possible values ​​in the data sequence to be compressed as the first and second boundary values, but uses a relatively long search time. This method guarantees the correctness of the separation and achieves minimal storage cost.

[0076] Furthermore, the step of traversing and selecting different values ​​in the sorted sequence as the first boundary value and the second boundary value to obtain multiple sets of boundary value combinations, and calculating the cost corresponding to each set of boundary value combinations, may include: determining the values ​​in the sorted sequence as the first boundary value in ascending order of value; after determining the first boundary value, determining the values ​​in the sorted sequence as the second boundary value in ascending order of value and value greater than the first boundary value, and calculating the corresponding cost to obtain the cost corresponding to each set of boundary value combinations.

[0077] For example, consider a time series containing 10 numbers (0, 124, 126, 125, 127, 128, 131, 130, 128, 255). The smallest number is 0, and the largest is 255. We can first sort the data to obtain the sequence (0, 124, 125, 126, 127, 128, 128, 130, 131, 255). Then, we iterate through all the possible values ​​to find the optimal boundary. For example, the smaller boundary could be set to 0, and the larger boundary could be set to 124, 125, 126, etc. The cost of each boundary selection can be calculated using the preliminary expression described below. After iterating through all possibilities, we compare and select the boundary value corresponding to the minimum cost. This iteration method uses two nested loops, resulting in an O(n) squared complexity.

[0078] For example, the preliminary expression could be:

[0079] Definition 1, Storage cost of bit packing:

[0080] (1), where, yes The maximum and minimum values ​​in the sequence, Indicates the length of the sequence.

[0081] In the separation algorithm, the key focus is on finding two boundary values. Using two boundary values, the original sequence can be divided into three categories:

[0082] Definition 2, Normal value: (2),

[0083] Definition 3, outliers: (3),

[0084] Definition 4, outliers: (4).

[0085] At this point, the storage cost of the bit pack with anomaly separation is divided into three parts, plus a fourth part for storing the location of the outlier, each part being similar to that in Definition 1.

[0086] Definition 5, Storage cost of packing bits with exception separation:

[0087]

[0088] in, Indicates the length of the outlier. Indicates the length of the outlier.

[0089] Definition 6, Cumulative counting, recording sequence medium to small and less than or equal to Number of numbers:

[0090] (6)

[0091] Definition 7, based on Definitions 5 and 6, the storage cost of packing abnormally separated bits can be re-expressed as:

[0092]

[0093] Understandably, methods that separate outliers based on precise value magnitude ensure the accuracy of the separation and achieve minimal storage costs.

[0094] For example, the pseudocode for method one is as follows:

[0095] ,

[0096] That is, the most basic approach is to iterate through the data to find the two optimal boundary values. .

[0097] The input is a sequence. The output is the optimal solution for the first and second boundary values. Step 1 indicates that... The sequence is sorted. Step 2 represents the ordered traversal of sequence X. Step 3 represents obtaining the count value based on Definition 6. Step 4 represents calculating a basic storage cost. Step 5 represents the ordered traversal of sequence X again as... The selection of values, step 6 represents the ordered traversal of the latter half of the X sequence as... Step 7 represents calculating the storage cost of this selection based on Definition 7; Step 8 represents comparing this cost with the optimal boundary scheme in the records; and Step 9 represents updating the lower boundary value in the optimal storage scheme. Step 10 represents updating the lower bound value in the optimal storage scheme. Step 11 represents updating the storage cost in the optimal storage scheme, and step 12 represents returning the optimal boundary value selection. .

[0098] Figure 3 This is an example of how the present invention separates and classifies sequence outliers based on precise value magnitude. For example... Figure 3 As shown, This indicates that the size range of the outlier is less than , This indicates that the normal value range is smaller than , This indicates that the size range of the outlier is less than .

[0099] Figure 4 This is a schematic diagram of the sequence sorting provided by the present invention. For example... Figure 4 As shown, firstly, the present invention sorts the sequence in ascending order, and obtains as follows: Figure 4 The cumulative count is shown. In this sequence, Algorithm 1 enumerates from the minimum value of 465 to the maximum value of 935. ,from Enumerate the next value up to the maximum value of 935. Finally, the algorithm finds the optimal solution (632, 696) with the minimum cost.

[0100] Method 2: Separate outliers in the sequence based on the precise bit width.

[0101] In an embodiment of the present invention, the target feature includes bit width. Determining the first boundary value and the second boundary value for dividing the data sequence to be compressed based on the target feature of each value in the data sequence to be compressed may include: determining the bit width of each value in the data sequence to be compressed; and determining the first boundary value and the second boundary value based on the bit width distribution corresponding to the bit width of each value.

[0102] It should be noted that, based on the method of determining the segmentation boundary according to the precise value magnitude, this invention proposes an efficient separation method based on bit width. In this scheme, using the bit width as the delimiter, the boundary value can be determined with less search time.

[0103] It is understandable that by using the above method of separating outliers based on precise bit width, the optimal solution for bit width separation can be obtained as the basis for data sequence separation, thus improving the separation efficiency.

[0104] Furthermore, determining the first boundary value and the second boundary value based on the bit width distribution corresponding to the bit width of each value may include: traversing each bit width within the first bit width range, then traversing and selecting all values ​​in the data sequence to be compressed as normal values, determining the target bit width and starting value corresponding to the normal values, wherein the target bit width is used to indicate the optimal bit width representing the normal values ​​in the data sequence to be compressed, and the starting value is used to indicate the optimal starting value representing the normal values ​​in the data sequence to be compressed; based on the target bit width and starting value, traversing and selecting all values ​​in the data sequence to be compressed as the starting value of the normal values, then traversing and selecting outliers and calculating the corresponding costs, wherein the costs are used to characterize the compression storage cost; and determining the value with the minimum corresponding cost as the first boundary value and the second boundary value.

[0105] It should be noted that the method of separating outliers in a sequence based on precise bit width size involves traversing the bit width range to find the optimal segmentation.

[0106] For example, consider a time series containing 10 numbers (0, 124, 126, 125, 127, 128, 131, 130, 128, 255). Two rounds of iteration are needed. In the first round, all possible bit widths for normal values ​​are iterated (ultimately selecting 3 bits), and then all values ​​are selected as possible starting points for normal values ​​(124). In the second round, the starting point for normal values ​​is selected first, and then the possible bit widths for outliers, i.e., excessively large values, are iterated. The optimal separation scheme is selected through these two rounds of iteration. The complexity of this approach is n log n.

[0107] Understandably, the precise bit-width separation method provided by this invention considers the specific bit widths of the center value and the outlier values ​​only within a short search time, thus improving search efficiency. By explaining all possible search cases, bit-width separation still returns the optimal solution as value separation. Correctness is ensured by transforming the solution based on value pairs from the set of sequence values.

[0108] For example, the pseudocode for separating sequence outliers based on precise bit width size is as follows:

[0109]

[0110] The input is a sequence. The output is the optimal solution for the first and second boundary values. Step 1 indicates that... The sequence is sorted. Steps 2 and 3 represent traversing the X sequence in order and obtaining the count value based on Definition 6. Step 4 represents calculating a basic storage cost. Steps 5 and 6 represent traversing the X sequence in order again as... The selected values, steps 7 and 8 represent the traversal. Determine the lower boundary value The selection process involves several steps: step 9 calculates the storage cost of the selected option based on definition 7; steps 10 to 13 compare this cost with the optimal boundary scheme in the records and update accordingly; and steps 14 and 15 iterate through the records. Determine the lower boundary value The selection process involves several steps. Step 16 calculates the storage cost of this selection based on Definition 7. Steps 17 to 20 compare this cost with the optimal boundary scheme in the records and update accordingly. Step 21 returns the optimal boundary value selection. .

[0111] First, after a rather complex mathematical discussion, it was found that only the separation cases shown in the following table need to be summarized:

[0112] Table 1 shows all possible cases of separation based on bit width.

[0113]

[0114] Similar to Algorithm 1, this invention calculates the cumulative count of values ​​in lines 1-3. Then, the algorithm enumerates each value in lines 5-13. And the cost for each corresponding β, where β ≤ γ. Based on the first case in Table II, the solution to consider is ( ,min + For the second case in Table II, the solution of the algorithm is in rows 14-20. Next, enumerate each And the cost of each γ.

[0115] Consider the sequence from before. First, we sort the sequence in ascending order and obtain the cumulative count, similar to Algorithm 1. In this sequence, Algorithm 2 enumerates from the minimum value of 465 to the maximum value of 935. For each = and β≤ γ, that is We consider = + The cost.

[0116] Figure 5 This is a schematic diagram illustrating a scenario for segmenting outliers in a sequence, as provided by the present invention. For example... Figure 5 As shown, for each = And β>γ, that is We consider As shown in the figure below. Finally, the algorithm found... = 632 and β = 6 are the optimal solutions.

[0117] Method 3: Separate outliers in the sequence based on the precise value size and bit width.

[0118] In an embodiment of the present invention, the target feature includes value size and bit width size. Determining the first boundary value and the second boundary value of the data sequence to be compressed based on the target feature of each value in the data sequence to be compressed may include: obtaining the median of the data sequence to be compressed; traversing and selecting normal values ​​with the median of the data sequence to be compressed as the center; determining the bit width range of the normal values; and determining the first boundary value and the second boundary value based on the bit width range of the normal values.

[0119] It should be noted that this invention proposes a strategy for performing approximate median separation in a very short time. Using this method, outlier separation can be completed in a relatively short time.

[0120] Furthermore, obtaining the median of the data sequence to be compressed may include: performing differential processing on each value in the data sequence to be compressed to obtain a processed data sequence; observing the normal distribution of each value in the processed data sequence; and determining the median based on the normal distribution of each value.

[0121] It should be noted that the method of using an approximate median separation algorithm to classify outliers in a sequence is similar to the method based on bit width separation. It assumes that the selection of normal values ​​is centered on the median of the sequence, searching within the bit width range of normal values, while those outside this range are outliers. In this case, only a one-dimensional search is needed, with a complexity of O(n).

[0122] For example, consider a time series containing 10 numbers (0, 124, 126, 125, 127, 128, 131, 130, 128, 255). In this example, the search can be understood as searching around 128.

[0123] Understandably, this invention proposes a method for approximate median separation, which considers the median in the separation process by taking into account the normal distribution of observed values, especially after differencing. Combined with the aforementioned bit width, an approximate separation can be determined in a very short time.

[0124] For example, the pseudocode for a method to separate sequence outliers based on precise value size and bit width is as follows:

[0125]

[0126] The input is a sequence. The output is the optimal solution for the first and second boundary values. Step 1 indicates that the optimal solution has been found. The sequence's median value; step 2 represents iterating through and selecting all values ​​in the sequence; step 3, up to step 10, is a branching structure, increasing the count of different buckets based on the relationship between the selected value and the sequence median; step 11 represents calculating a basic storage cost; and step 12 represents iterating through and selecting the bit width of normal values. Steps 13 and 14 represent updating the count of upper and lower outliers in the loop, and steps 15 and 16 represent the bit width based on the normal values. Given the determined upper and lower boundary values, step 17 determines the storage cost at this point based on Definition 5. Steps 18 to 21 compare this cost with the optimal boundary scheme in the records and update accordingly. Step 22 returns the optimal boundary value selection. .

[0127] Figure 6 This is a schematic diagram of the cumulative counting provided by the present invention. For example... Figure 6As shown, let's still consider the previous sequence. First, let's consider the previous sequence again. First, Algorithm 3 finds the median. = 674. Given a maximum value β = 9, it divides the values ​​into 19 buckets. The algorithm then searches for the bit width β with the minimum cost by enumerating β from 9 to 1. It returns a solution (610, 738) for β = 6. Figure 6 As shown, we only need to expand the range of several integer powers of 2 outwards from the median to make a selection.

[0128] In summary, Method 1 is a relatively direct and crude implementation, which involves iterating through the upper and lower bounds of outliers to obtain the most accurate partitioning results. Method 2 is an optimization that uses bit width to complete the partitioning by iterating through the upper and lower outliers, which greatly improves the speed while keeping the compression ratio constant. Method 3 is an approximate algorithm, with a compression ratio that is not as high as the second method but not by much, while the compression speed is much faster, achieving the shortest compression and decompression time.

[0129] In some embodiments, after determining the normal values ​​and obtaining the normal value sequence, the minimum bit width can be determined based on the normal values ​​for compression.

[0130] Figure 7 This is a flowchart illustrating the method for compressing at least two data sequences provided by the present invention. Figure 7 As shown, the at least two data sequences include a target data sequence, the target data sequence includes the normal value, and the data compression of the at least two data sequences respectively may include:

[0131] Step 301: Determine the minimum bit width required to store the normal value based on the range of the normal value.

[0132] Step 302: Store the normal value using the minimum bit width, where the first value stored using the minimum bit width is the minimum value among the normal values, and the last value stored using the minimum bit width is the maximum value among the normal values.

[0133] It should be noted that after determining the range of each sequence, they can be stored one by one using the minimum bit width, which is also part of the system design.

[0134] For example, consider a time series containing 10 numbers (0, 124, 126, 125, 127, 128, 131, 130, 128, 255), where the smallest is 0 and the largest is 255. Normally, each number would require 8 bits (2^8 = 256) to store. However, most (8) normal values ​​fall within the range of 124 to 131, with only 8 possible values, requiring only 3 bits (2^3) for storage. That is, 000 represents 124, 001 represents 125, 010 represents 126, 011 represents 127, and so on. This uses three bits to store each normal value. This correspondence only requires storing the upper and lower bounds. Simultaneously, the range of outliers can be determined to be a smaller range, thus also requiring a smaller bit width for storage.

[0135] Understandably, after identifying outliers, this invention proposes an optimized bit-stuffing data compression method. Specifically, this invention proposes a bit-stuffing method based on outlier separation, which improves bit stuffing by separating the upper outlier sequence and the lower outlier sequence. This method can further reduce the cost of compressed storage.

[0136] The method for storing upper and lower outlier sequences provided by this invention, while having a narrow extension (i.e., a compressed bit width) for the remaining center values, requires additional overhead to represent the positions of the separated outliers. Therefore, employing a better-defined threshold to separate upper and lower outliers results in lower storage costs. This is achieved instead of enumerating all possible values ​​as separators for upper and lower outliers, which would be excessively time-consuming.

[0137] It is understood that the anomaly-based data compression algorithm of this invention is compatible with any existing compression method that uses bit packing, and can replace bit-stuffing blocks in various algorithms after deployment. Extensive experiments on many real datasets show that, among various compression methods, the compression ratio of the replaced data compression algorithm is significantly improved from around 2.75 to 3.25.

[0138] The following describes an exemplary application of the embodiments of the present invention in a practical application scenario.

[0139] by Figure 3 For example. Figure 3 As shown, Xmax represents the upper outlier, Xu represents the normal value, and Xmin represents the lower outlier. Indicates the first boundary value. This represents the second boundary value. The diagram was obtained through anomaly separation, demonstrating the method of determining anomaly separation based on two boundary values, which then guides subsequent bit packing and compression.

[0140] based on Figure 3The data sequence to be compressed is divided into three parts: a normal value sequence and an upper / lower outlier value sequence. The range of a large number of normal values ​​is restricted to a relatively small range, which greatly reduces the bit width required to store them.

[0141] Furthermore, to verify the anomaly-based data compression method provided by this invention, its deployment was implemented, and extensive experiments were conducted on numerous real-world datasets. Experiments show that the compression ratio is significantly improved from approximately 2.75 for existing methods to an average of 3.25 for the precise bit-width separation method. Therefore, it is strongly recommended that the anomaly-based data compression method proposed in this invention replace simple bit stuffing in practice.

[0142] Based on the foregoing embodiments, this invention provides a data compression device based on anomaly separation. The modules and units included in the device can be implemented by a processor; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0143] The anomaly-based data compression device provided by the present invention is described below. The anomaly-based data compression device described below and the anomaly-based data compression method described above can be referred to in correspondence.

[0144] Figure 8 This is a schematic diagram of the data compression device based on anomaly separation provided by the present invention. Figure 8 As shown, the device 500 includes a data partitioning module 501 and a data compression module 502, wherein:

[0145] The data partitioning module 501 is used to divide the data sequence to be compressed into at least two data sequences based on the normal values ​​and outliers in the data sequence to be compressed. One of the data sequences contains the normal values, and the other data sequences contain the outliers. The outliers are used to indicate one or more values ​​whose values ​​deviate significantly from the values ​​of most of the values ​​in the data sequence to be compressed. The number of normal values ​​is greater than the number of outliers.

[0146] The data compression module 502 is used to compress the at least two data sequences respectively.

[0147] In some embodiments, the data partitioning module 501 includes a boundary determination unit and a data partitioning unit, wherein,

[0148] The boundary determination unit is used to determine a first boundary value and a second boundary value for dividing the data sequence to be compressed based on the target features of each value in the data sequence to be compressed. The target features include value size and / or bit width. The first boundary value is smaller than the second boundary value. The bit width is used to indicate the number of bits required to store the binary value of the value.

[0149] The data partitioning unit is used to divide the data sequence to be compressed into an upper outlier sequence, a normal value sequence, and a lower outlier sequence according to the first boundary value and the second boundary value. The upper outlier sequence includes values ​​in the data sequence to be compressed that are less than the first boundary value. The lower outlier sequence includes values ​​in the data sequence to be compressed that are greater than the second boundary value. The normal value sequence includes values ​​in the data sequence to be compressed that are greater than or equal to the first boundary value and less than or equal to the second boundary value.

[0150] In some embodiments, the target feature includes a value magnitude, and the boundary determination unit includes a data sorting component, a cost calculation component, and a first determination component, wherein,

[0151] The data sorting component is used to sort the values ​​in the data sequence to be compressed according to their size, so as to obtain a sorted sequence.

[0152] The cost calculation component is used to traverse and select different values ​​as the first boundary value and the second boundary value in the sorted sequence to obtain multiple sets of boundary value combinations, and calculate the cost corresponding to each set of boundary value combinations. The cost is used to characterize the compressed storage cost.

[0153] The first determining component is used to determine the values ​​in the corresponding combination of boundary values ​​with the lowest cost as the first boundary value and the second boundary value.

[0154] In some embodiments, the cost calculation component is specifically used to: determine the values ​​in the sorted sequence as the first boundary value in ascending order of value; after determining the first boundary value, determine the values ​​in the sorted sequence as the second boundary value in ascending order of value and value greater than the first boundary value, and calculate the corresponding cost to obtain the cost corresponding to each set of boundary value combinations.

[0155] In some embodiments, the target feature includes a bit width, and the boundary determination unit includes a bit width determination component and a second determination component, wherein...

[0156] The bit width determining component is used to determine the bit width of each value in the data sequence to be compressed;

[0157] The second determining component is used to determine the first boundary value and the second boundary value based on the bit width distribution corresponding to the bit width of each value.

[0158] In some embodiments, the second determining component is specifically used to: traverse each bit width within the first bit width range, then traverse and select all values ​​in the data sequence to be compressed as normal values, determine the target bit width and starting value corresponding to the normal value, wherein the target bit width is used to indicate the optimal bit width representing the normal value in the data sequence to be compressed, and the starting value is used to indicate the optimal starting value representing the normal value in the data sequence to be compressed.

[0159] Based on the target bit width and the starting value, all values ​​in the data sequence to be compressed are selected as the starting value of the normal value. Then, outliers are selected and the corresponding costs are calculated. The costs are used to characterize the compression storage cost.

[0160] The values ​​that minimize the corresponding costs are determined as the first boundary value and the second boundary value.

[0161] In some embodiments, the target feature includes value size and bit width size, and the boundary determination unit includes a median determination component, a normal value determination component, and a third determination component, wherein,

[0162] The median determination component is used to obtain the median of the data sequence to be compressed;

[0163] The normal value determination component is used to traverse and select normal values ​​centered on the median of the data sequence to be compressed.

[0164] The third determining component is used to determine the bit width range of the normal value, and to determine the first boundary value and the second boundary value based on the bit width range of the normal value.

[0165] In some embodiments, the median determination component is specifically used to: perform differential processing on each value in the data sequence to be compressed to obtain a processed data sequence; observe the normal distribution of each value in the processed data sequence; and determine the median based on the normal distribution of each value.

[0166] In some embodiments, the at least two data sequences include a target data sequence, the target data sequence includes the normal values, and the data compression module is specifically used to: determine the minimum bit width required to store the normal values ​​based on the number of normal values;

[0167] The normal value is stored using the minimum bit width, where the first value stored using the minimum bit width is the starting value of the normal value, and the last value stored using the minimum bit width is the ending value of the normal value.

[0168] In this embodiment of the invention, a method is provided to divide the data sequence to be compressed into at least two data sequences based on normal and outlier values ​​in the sequence, and to compress the at least two data sequences separately. This achieves the effect of separating and compressing outliers for storage, reducing the time and space costs of data compression, and improving data compression efficiency.

[0169] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. For example... Figure 9 As shown, the electronic device 600 may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can invoke logical instructions in the memory 630 to execute a data compression method based on anomaly separation. This method includes: dividing the data sequence to be compressed into at least two data sequences according to normal values ​​and anomalies in the data sequence to be compressed; one of the at least two data sequences contains the normal values, and the other data sequences contain the anomalies; the anomalies indicate one or more values ​​whose values ​​significantly deviate from the values ​​of most values ​​in the data sequence to be compressed; the number of normal values ​​is greater than the number of anomalies; and compressing the at least two data sequences separately.

[0170] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the data compression method based on anomaly separation provided by the above methods. The method includes: dividing the data sequence to be compressed into at least two data sequences according to normal values ​​and anomalies in the data sequence to be compressed, wherein one of the at least two data sequences contains the normal values ​​and the other data sequences contain the anomalies, the anomalies being used to indicate one or more values ​​whose value size deviates significantly from the value size of most values ​​in the data sequence to be compressed, and the number of normal values ​​is greater than the number of anomalies; and performing data compression on the at least two data sequences respectively.

[0172] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0173] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the data compression method based on anomaly separation provided by the above methods. The method includes: dividing the data sequence to be compressed into at least two data sequences according to normal values ​​and anomalies in the data sequence to be compressed, wherein one of the at least two data sequences contains the normal values ​​and the other data sequences contain the anomalies, the anomalies being used to indicate one or more values ​​whose value size deviates significantly from the value size of most values ​​in the data sequence to be compressed, and the number of normal values ​​is greater than the number of anomalies; and performing data compression on the at least two data sequences respectively.

[0174] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0175] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0176] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, radio frequency (RF), etc., or any suitable combination thereof.

[0177] Computer program code for performing the operations described herein can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data compression method based on anomaly separation, characterized in that, include: Based on the normal values ​​and outliers in the data sequence to be compressed, the data sequence to be compressed is divided into at least two data sequences. One of the at least two data sequences contains the normal values, and the other data sequences contain the outliers. The outliers are used to indicate one or more values ​​whose values ​​deviate significantly from the values ​​of most of the values ​​in the data sequence to be compressed. The number of normal values ​​is greater than the number of outliers. Compress the data into the at least two data sequences respectively; The step of dividing the data sequence to be compressed into at least two data sequences based on normal and outlier values ​​in the data sequence to be compressed includes: The first boundary value and the second boundary value are determined based on the target features of each value in the data sequence to be compressed. The target features include the value size and / or the bit width. The first boundary value is smaller than the second boundary value. The bit width is used to indicate the number of bits required to store the binary value of the value. The data sequence to be compressed is divided into an upper outlier sequence, a normal value sequence, and a lower outlier sequence based on the first boundary value and the second boundary value. The upper outlier sequence includes values ​​in the data sequence to be compressed that are less than the first boundary value. The lower outlier sequence includes values ​​in the data sequence to be compressed that are greater than the second boundary value. The normal value sequence includes values ​​in the data sequence to be compressed that are greater than or equal to the first boundary value and less than or equal to the second boundary value. The target features include value size and bit width. Determining the first and second boundary values ​​for dividing the data sequence to be compressed based on the target features of each value in the data sequence to be compressed includes: Obtain the median of the data sequence to be compressed; The normal values ​​are selected by iterating through the data sequence to be compressed, with the median as the center. Determine the bit width range of the normal value, and determine the first boundary value and the second boundary value based on the bit width range of the normal value.

2. The data compression method based on anomaly separation according to claim 1, characterized in that, The target feature includes value magnitude. Determining the first boundary value and the second boundary value for dividing the data sequence to be compressed based on the target features of each value in the data sequence to be compressed includes: The values ​​in the data sequence to be compressed are sorted according to their size to obtain a sorted sequence; In the sorted sequence, different values ​​are selected as the first boundary value and the second boundary value to obtain multiple sets of boundary value combinations. The cost corresponding to each set of boundary value combinations is calculated. The cost is used to characterize the compressed storage cost. The values ​​in the boundary value combination that minimizes the corresponding cost are determined as the first boundary value and the second boundary value.

3. The data compression method based on anomaly separation according to claim 2, characterized in that, The process involves iterating through the sorted sequence, selecting different values ​​as the first and second boundary values ​​to obtain multiple sets of boundary value combinations, and calculating the cost corresponding to each set of boundary value combinations, including: The values ​​in the sorted sequence are determined as the first boundary values ​​in ascending order of their values. After determining the first boundary value, the values ​​in the sorted sequence are then determined as the second boundary values ​​in order of ascending value and value greater than the first boundary value, and the corresponding costs are calculated to obtain the costs corresponding to each combination of boundary values.

4. The data compression method based on anomaly separation according to claim 1, characterized in that, The target feature includes bit width. Determining the first and second boundary values ​​for dividing the data sequence to be compressed based on the target features of each value in the data sequence to be compressed includes: Determine the bit width of each value in the data sequence to be compressed; The first boundary value and the second boundary value are determined based on the bit width distribution corresponding to the bit width of each value.

5. The data compression method based on anomaly separation according to claim 4, characterized in that, The step of determining the first boundary value and the second boundary value based on the bit width distribution corresponding to the bit width of each value includes: Iterate through each bit width within the first bit width range, then iterate through and select all values ​​in the data sequence to be compressed as normal values, determine the target bit width and starting value corresponding to the normal value, the target bit width is used to indicate the optimal bit width representing the normal value in the data sequence to be compressed, and the starting value is used to indicate the optimal starting value representing the normal value in the data sequence to be compressed; Based on the target bit width and the starting value, all values ​​in the data sequence to be compressed are selected as the starting value of the normal value. Then, outliers are selected and the corresponding costs are calculated. The costs are used to characterize the compression storage cost. The values ​​that minimize the corresponding costs are determined as the first boundary value and the second boundary value.

6. The data compression method based on anomaly separation according to claim 1, characterized in that, Obtaining the median of the data sequence to be compressed includes: Perform differential processing on each value in the data sequence to be compressed to obtain the processed data sequence; Observe the normal distribution of each value in the processed data sequence; The median is determined based on the normal distribution of the values.

7. The data compression method based on anomaly separation according to claim 1, characterized in that, The at least two data sequences include a target data sequence, the target data sequence includes the normal values, and the data compression of the at least two data sequences includes: The minimum bit width required to store the normal values ​​is determined based on the number of normal values. The normal value is stored using the minimum bit width, where the first value stored using the minimum bit width is the starting value of the normal value, and the last value stored using the minimum bit width is the ending value of the normal value.

8. A data compression device based on anomaly separation, characterized in that, include: The data partitioning module is used to divide the data sequence to be compressed into at least two data sequences based on normal values ​​and outliers in the data sequence to be compressed. One of the data sequences contains the normal values, and the other data sequences contain the outliers. The outliers are used to indicate one or more values ​​whose values ​​deviate significantly from the values ​​of most values ​​in the data sequence to be compressed. The number of normal values ​​is greater than the number of outliers. A data compression module is used to compress the at least two data sequences respectively; The data partitioning module includes: A boundary determination unit is used to determine a first boundary value and a second boundary value for dividing the data sequence to be compressed based on the target features of each value in the data sequence to be compressed. The target features include value size and / or bit width. The first boundary value is smaller than the second boundary value. The bit width is used to indicate the number of bits required to store the binary value of the value. A data partitioning unit is used to divide the data sequence to be compressed into an upper outlier sequence, a normal value sequence, and a lower outlier sequence according to the first boundary value and the second boundary value. The upper outlier sequence includes values ​​in the data sequence to be compressed that are less than the first boundary value. The lower outlier sequence includes values ​​in the data sequence to be compressed that are greater than the second boundary value. The normal value sequence includes values ​​in the data sequence to be compressed that are greater than or equal to the first boundary value and less than or equal to the second boundary value. The target feature includes bit width, and the boundary determination unit includes: Bit width determination component, used to determine the bit width of each value in the data sequence to be compressed; The second determining component is used to determine the first boundary value and the second boundary value based on the bit width distribution corresponding to the bit width of each value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data compression method based on anomaly separation as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data compression method based on anomaly separation as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data compression method based on anomaly separation as described in any one of claims 1 to 7.

Images