Adaptive data compression method, system and device

Abstract

The present application discloses an adaptive data compression method, system, and device, relating to the field of data compression and information processing technology. The method comprises: calculating the Shannon entropy and repetition rate of the data to be compressed; dividing the data to be compressed into a first half and a second half according to the data length, and calculating the Shannon entropy of the first half of the data to be compressed and the Shannon entropy of the second half of the data to be compressed; determining a target data compression algorithm based on the Shannon entropy of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy of the first half of the data to be compressed, the Shannon entropy of the second half of the data to be compressed, and an adaptive compression algorithm determination model; and compressing the data to be compressed using the target data compression algorithm to obtain an optimal compression codeword sequence. The present application improves the compression efficiency of the data to be compressed and the universality of the compression method.

Description

Technical Field

[0001] The present application relates to the technical field of data compression and information processing, and in particular to an adaptive data compression method, system and device. Background Art

[0002] With the rapid development of information technology, data volumes are constantly increasing, and the demand for data compression algorithms is growing. Traditional data compression algorithms, such as Huffman coding and arithmetic coding, often face problems such as low compression efficiency and poor adaptability to different data types when processing large amounts of complex data. This is especially true when processing complex data with high entropy or strong repetitiveness. Existing compression methods often fail to achieve the desired compression ratio, resulting in poor compression results.

[0003] Adaptive compression, as an emerging compression technology, can dynamically adjust the compression method based on the different characteristics of the data, improving compression efficiency and decompression speed. However, traditional adaptive compression methods still have certain limitations in method selection and optimization, making it difficult to fully meet the needs of different data types. Therefore, there is an urgent need for a data compression method based on adaptive methods that can flexibly select compression methods based on different data characteristics, thereby improving compression rate while reducing computational complexity. Summary of the Invention

[0004] The purpose of this application is to provide an adaptive data compression method, system and device to improve the compression efficiency of the data to be compressed and the universality of the compression method.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an adaptive data compression method, the adaptive data compression method comprising:

[0007] Calculate the Shannon entropy and repetition rate of the data to be compressed; the repetition rate of the data to be compressed is the probability of occurrence of the most frequent character in the data to be compressed;

[0008] Divide the data to be compressed into a first half and a second half according to the data length, and calculate the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed;

[0009] A target data compression algorithm is determined based on the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, the Shannon entropy value of the second half of the data to be compressed, and an adaptive compression algorithm determination model; the adaptive compression algorithm determination model is obtained by training a feedforward fully connected neural network model using a training set, and the training set includes: the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, the Shannon entropy value of the second half of the sample data, and categories of multiple data compression algorithms;

[0010] The target data compression algorithm is used to compress the data to obtain the optimal compression codeword sequence.

[0011] In one embodiment, calculating the Shannon entropy and repetition rate of the data to be compressed specifically includes:

[0012] Calculate the number of occurrences of each character in the data to be compressed;

[0013] Based on the number of occurrences of each character and the length of the data to be compressed, the occurrence probability of the most frequent character in the data to be compressed is determined, and the occurrence probability of the most frequent character is used as the repetition rate of the data to be compressed;

[0014] The Shannon entropy value of the data to be compressed is calculated based on the occurrence probability of each character in the data to be compressed.

[0015] In one embodiment, determining a target data compression algorithm based on the Shannon entropy of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy of the first half of the data to be compressed, the Shannon entropy of the second half of the data to be compressed, and an adaptive compression algorithm determination model specifically includes:

[0016] Inputting the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, and the Shannon entropy value of the second half of the data to be compressed into the adaptive compression algorithm determination model, and outputting the optimal category;

[0017] The data compression algorithm corresponding to each optimal category is determined as the target data compression algorithm.

[0018] In one embodiment, there are one or more optimal categories, and one optimal category corresponds to one target data compression algorithm.

[0019] In one embodiment, the target data compression algorithm is used to compress the data to obtain an optimal compression codeword sequence, specifically including:

[0020] When the target data compression algorithm is one, the target data compression algorithm is used to compress the data to be compressed to obtain the optimal compression codeword sequence;

[0021] When there are multiple target data compression algorithms, the data to be compressed are compressed using the multiple target data compression algorithms to obtain compression codeword sequences corresponding to the multiple target data compression algorithms;

[0022] Determine the compression ratio of the compression codeword sequence corresponding to each target data compression algorithm;

[0023] The compression codeword sequence corresponding to the target data compression algorithm with the highest compression rate is determined as the optimal compression codeword sequence.

[0024] In one embodiment, the training process of the adaptive compression algorithm determination model specifically includes:

[0025] Construct a training set;

[0026] Build a feedforward fully connected neural network model;

[0027] Taking the Shannon entropy value of sample data, the repetition rate of sample data, the Shannon entropy value of the first half of the sample data and the Shannon entropy value of the second half of the sample data as input, and the data compression algorithm as output, the feedforward fully connected neural network model is iteratively trained until the number of iterations reaches the maximum value or the loss function reaches the minimum value, and the iterative training is stopped to obtain the adaptive compression algorithm determination model.

[0028] In one embodiment, the data compression algorithm includes: a standard LZW compression algorithm, a reset LZW compression algorithm, a block LZW compression algorithm, a sliding window LZW compression algorithm, and a threshold-based dictionary reset LZW compression algorithm.

[0029] In one embodiment, the adaptive data compression method further includes:

[0030] Decompress the optimal compressed codeword sequence using a corresponding data decompression algorithm to obtain decompressed data;

[0031] Compare the decompressed data with the data to be compressed to see if they are consistent;

[0032] If yes, the optimal compression codeword sequence is used as the final compression result of the data to be compressed;

[0033] If not, the model parameters of the adaptive compression algorithm determination model are updated, and the updated adaptive compression algorithm determination model is used to determine the target data compression algorithm.

[0034] In a second aspect, the present application provides an adaptive data compression system, which is used to implement the adaptive data compression method. The adaptive data compression system includes:

[0035] The first calculation unit is used to calculate the Shannon entropy value and repetition rate of the data to be compressed; the repetition rate of the data to be compressed is the probability of occurrence of the most frequent character in the data to be compressed;

[0036] A second calculation unit is used to divide the data to be compressed into a first half and a second half according to the data length, and calculate the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed;

[0037] a target data compression algorithm determination unit, configured to determine a target data compression algorithm based on a Shannon entropy value of the data to be compressed, a repetition rate of the data to be compressed, a Shannon entropy value of the first half of the data to be compressed, a Shannon entropy value of the second half of the data to be compressed, and an adaptive compression algorithm determination model; the adaptive compression algorithm determination model is obtained by training a feedforward fully connected neural network model using a training set, the training set including: the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, the Shannon entropy value of the second half of the sample data, and categories of multiple data compression algorithms;

[0038] The optimal compression codeword sequence determining unit is used to compress the data to be compressed using a target data compression algorithm to obtain an optimal compression codeword sequence.

[0039] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described adaptive data compression methods.

[0040] According to the specific embodiments provided in this application, this application has the following technical effects:

[0041] The present application discloses an adaptive data compression method, system, and device. First, the Shannon entropy and repetition rate of the data to be compressed are calculated; second, the data to be compressed is divided into a first half and a second half according to the data length, and the Shannon entropy of the first half of the data to be compressed and the Shannon entropy of the second half of the data to be compressed are calculated; further, based on the Shannon entropy of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy of the first half of the data to be compressed, the Shannon entropy of the second half of the data to be compressed, and an adaptive compression algorithm determination model, a target data compression algorithm is determined; finally, the target data compression algorithm is used to compress the data to be compressed to obtain an optimal compression codeword sequence. By establishing an adaptive compression algorithm determination model, the present application can adaptively select the optimal data compression method that matches the data to be compressed, thereby improving data compression efficiency; the method is applicable to the compression of various types of data, thereby improving the universality of the data compression method. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A schematic diagram of a flow chart of an adaptive data compression method provided in one embodiment of the present application;

[0044] Figure 2 A schematic diagram of the training process provided in one embodiment of the present application;

[0045] Figure 3 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0047] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0048] In an exemplary embodiment, Figure 1 As shown, an adaptive data compression method is provided, comprising the following steps.

[0049] Step S1, calculating the Shannon entropy value and repetition rate of the data to be compressed; the repetition rate of the data to be compressed is the probability of occurrence of the most frequent character in the data to be compressed.

[0050] As an optional implementation, step S1 specifically includes:

[0051] Step S11, calculating the number of occurrences of each character in the data to be compressed.

[0052] Step S12: Based on the number of occurrences of each character and the length of the data to be compressed, the probability of occurrence of the most frequent character in the data to be compressed is determined, and the probability of occurrence of the most frequent character is used as the repetition rate of the data to be compressed. The probability of occurrence of each character is calculated using the following formula:

[0053] p i = ni / N(1)

[0054] in, p i Indicates the i The probability of occurrence of characters; n i Indicates the i The number of occurrences of characters; N represents the data length.

[0055] Step S13, based on the occurrence probability of each character in the data to be compressed, calculate the Shannon entropy value of the data to be compressed. The calculation formula of the Shannon entropy value of the data to be compressed is as follows:

[0056] H=−∑ p i log2( p i )(2)

[0057] Where H is the Shannon entropy of the data to be compressed.

[0058] Specifically, first, the entire segment of data to be compressed is taken as input. In order to count the number of times each character in the data to be compressed is created, an empty frequency dictionary is first established, and then each character in the data to be compressed is scanned in turn. Every time a character is encountered, the character count in the dictionary is increased by one. After the character count is completed, the total data length N (that is, the total number of characters) and the number of times each character appears nᵢ are obtained.

[0059] Next, calculate the probability of each character p i , and calculate the Shannon entropy value H accordingly. At the same time, compare the size of the probability of occurrence of all characters, and take the largest probability value (that is, the probability of occurrence of the most frequent character) as the repetition rate of the data to be compressed R max Among them, the Shannon entropy value is used to evaluate the randomness of the data to be compressed, and the repetition rate is used to evaluate the repeatability of the data to be compressed.

[0060] Step S2: Divide the data to be compressed into a first half and a second half according to the data length, and calculate the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed.

[0061] The calculation formulas for the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed are shown in formula (2).

[0062] Step S3, determining the target data compression algorithm based on the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, the Shannon entropy value of the second half of the data to be compressed, and the adaptive compression algorithm determination model; the adaptive compression algorithm determination model is obtained by training a feedforward fully connected neural network model using a training set, and the training set includes: the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, the Shannon entropy value of the second half of the sample data, and the categories of multiple data compression algorithms.

[0063] As an optional implementation, in step S3, the data compression algorithm includes: a standard LZW (short for Lempel-Ziv-Welch) compression algorithm, a reset LZW compression algorithm, a block LZW compression algorithm, a sliding window LZW compression algorithm, and a threshold-based dictionary reset LZW compression algorithm.

[0064] As an optional implementation, in step S3, the adaptive compression algorithm determines the training process of the model, specifically including:

[0065] Step S300: construct a training set.

[0066] Step S301: construct a feedforward fully connected neural network model.

[0067] Step S302, using the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, and the Shannon entropy value of the second half of the sample data as input, and the data compression algorithm as output, iteratively train the feedforward fully connected neural network model until the number of iterations reaches the maximum or the loss function reaches the minimum, stop iterative training, and obtain the adaptive compression algorithm to determine the model. The training process diagram is shown in the figure below. Figure 2 shown.

[0068] Among them, when training the adaptive compression algorithm to determine the model, the Shannon entropy value, repetition rate, Shannon entropy value of the first half of the data, and Shannon entropy value of the second half of the data are used as input features. In addition, the entropy difference (that is, the difference between the Shannon entropy value of the first half of the sample data and the Shannon entropy value of the second half of the sample data) can also be added to accelerate the convergence of the adaptive compression algorithm to determine the model.

[0069] Specifically, during model training, the optimal data compression algorithm is adaptively selected based on preset rules. The preset rules are as follows:

[0070] 1) If the overall Shannon entropy of the data to be compressed is higher than the preset Shannon entropy threshold, and the repetition rate (i.e., the probability of occurrence of the most frequent characters) is lower than the preset probability threshold, indicating that the data is relatively random and lacks obvious repetitiveness, the standard LZW compression algorithm is selected.

[0071] 2) If the length of the data to be compressed is less than the length threshold, and the overall Shannon entropy value of the data to be compressed is lower than the preset Shannon entropy threshold or the repetition rate is higher than the probability threshold, the LZW compression algorithm is reset to efficiently utilize the repetitive pattern in short data and avoid the overhead of complex methods.

[0072] 3) If the length of the data to be compressed is greater than the length threshold and the overall Shannon entropy value is lower than the preset Shannon entropy threshold (which means that the data is long and has high repetitiveness or redundancy overall), the threshold-based dictionary reset LZW compression algorithm is selected to prevent excessive dictionary growth and improve overall compression efficiency.

[0073] 4) If the structural strength of the data to be compressed exceeds the preset strength threshold or the repetition rate is higher than the probability threshold, the block LZW compression algorithm is selected.

[0074] Specifically, if the structural strength of the data to be compressed exceeds a preset strength threshold, the data is considered to have strong structure; otherwise, it is considered to have weak structure. The key to determining the strength of the "structural strength" of the data to be compressed lies in identifying whether there are reusable stable patterns between and within windows. Specifically, the data to be compressed is divided into multiple windows of fixed length. If the average intersection-of-two ratio (IoU) of shared substrings between adjacent windows is no less than a preset IoU threshold, or if high-frequency repetition of header / trailer bytes is found at fixed offsets within each window, the block LZW compression algorithm is selected.

[0075] In practice, the data to be compressed is first divided into a series of continuous windows of fixed length. Within each window, several substrings of moderate length are then extracted. These substrings are then used to measure the similarity between adjacent windows. If more than half of the substrings recur in adjacent windows, the content of each window is highly templated, indicating strong structure. Furthermore, the byte distribution is analyzed at fixed offsets within each window. If a large number of windows share the same header or trailer bytes at the same offset, and these "template bits" reach eight or more, or account for more than 10 percent of the window bytes, this also indicates that the data to be compressed has a clear protocol frame structure. For data that meets any of these conditions, the block-based LZW compression algorithm allows each window to maintain its own dictionary, preventing differences between blocks from diluting the compression effect and fully leveraging structural information to improve compression. If neither test result is significant, the data is considered weakly structured.

[0076] 5) If the data to be compressed has strong local dependence or temporal variability, choose the sliding window LZW compression algorithm.

[0077] Specifically, if the conditional entropy of the characters in the short distance of the data to be compressed is lower than the preset conditional entropy threshold, and the Kullback-Leibler (KL) divergence of adjacent windows is lower than the preset smoothing threshold, the sliding window LZW compression algorithm is selected to dynamically capture and utilize the local repetitive patterns of the data.

[0078] Among them, "local dependence or temporal variability" means that the same data stream can be predicted with high precision within a very short byte interval, and this predictable statistical law will drift slowly over time; in order to quantify this phenomenon, the entire segment of compressed data is first divided into continuous windows according to a fixed number of bytes (for example, 256 bytes, 512 bytes or 1 KB), and then the joint occurrence frequency of adjacent or neighboring bytes (for example, 1 to 3 byte intervals) is counted in each window to calculate the average conditional entropy within the window; when the conditional entropy is significantly lower than the preset conditional entropy threshold (for example, 3.5 bits, which is much lower than the maximum uncertainty of a single byte of 8 bits), it means that the "short distance" predictability is very strong, that is, the local dependence is significant; then, the Kullback-Leibler divergence of the overall character distribution of the two adjacent windows is calculated and smoothed. If this divergence is lower than the preset smoothing threshold (for example, 0.10 to 0.15) for a long time, it means that the statistical distribution evolves very slowly over time and presents a trend of Only when low conditional entropy and low KL divergence are observed at the same time, can we determine that the data has strong local dependence and smooth time series changes, and choose the sliding window LZW compression algorithm, because at this time the dictionary established in the previous window still has a high hit rate in the next window, and the old entries that gradually slide out can make room for the gradual new pattern; on the contrary, if the conditional entropy does not drop to the threshold or the KL divergence frequently exceeds the limit, it means that either the local predictability is insufficient or the pattern changes are too drastic, and the sliding window LZW compression algorithm will hardly bring advantages. Other data compression methods can be selected based on global indicators such as entropy value, repetition rate and data length.

[0079] All thresholds are initially determined through cross-validation on typical data sets and written back to the feature library along with the compression effects during operation, allowing the neural network to adjust the model parameters online to ensure that the judgment criteria can be automatically updated for new types of data, thereby achieving open, sufficient and implementable local dependency detection and algorithm selection.

[0080] During training, a corresponding compression ratio threshold is set. The compression results and compression ratios of the compression algorithms whose compression ratios exceed the compression ratio threshold are stored in a compression feature library. Combined with the compression feature library, pattern recognition techniques are used to process subsequent data to obtain an adaptive compression algorithm determination model, improving data compression efficiency and decompression accuracy. Subsequently, the compression ratio of each data compression algorithm is fed back to the compression feature library through a feedback mechanism, thereby improving the selection of compression algorithms for subsequent data. This allows the adaptive compression algorithm determination model to dynamically adjust the optimal data compression algorithm based on the latest data characteristics, optimizing compression ratio and decompression performance, and improving the adaptability and compression efficiency of the adaptive compression algorithm determination model.

[0081] As an optional implementation, step S3 specifically includes:

[0082] Step S31 , inputting the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, and the Shannon entropy value of the second half of the data to be compressed into an adaptive compression algorithm determination model, and outputting the optimal category.

[0083] Step S32: Determine the data compression algorithm corresponding to each optimal category as the target data compression algorithm.

[0084] As an optional implementation, in step S31, there are one or more optimal categories, and one optimal category corresponds to one target data compression algorithm.

[0085] Step S4: compress the data to be compressed using a target data compression algorithm to obtain an optimal compression codeword sequence.

[0086] As an optional implementation, step S4 specifically includes:

[0087] Step S41: When the target data compression algorithm is one, the target data compression algorithm is used to compress the data to be compressed to obtain an optimal compression codeword sequence.

[0088] Step S42: When there are multiple target data compression algorithms, the data to be compressed is compressed using the multiple target data compression algorithms respectively to obtain compression codeword sequences corresponding to the multiple target data compression algorithms.

[0089] Step S43: Determine the compression ratio of the compression codeword sequence corresponding to each target data compression algorithm.

[0090] Step S44: Determine the compression codeword sequence corresponding to the target data compression algorithm with the highest compression ratio as the optimal compression codeword sequence.

[0091] Specifically, when the data to be compressed meets more than two preset rules at the same time, it indicates that the data to be compressed has complex or compound structural characteristics. The data to be compressed is first compressed according to each rule and the compression rate is calculated. Finally, the compression codeword sequence corresponding to the data compression algorithm with the highest compression rate is selected as the optimal compression codeword sequence.

[0092] As an optional implementation, in step S4, the compression processing includes: updating the dictionary and generating corresponding codewords through character-by-character matching. When the dictionary reaches the maximum capacity, resetting or blocking operations are performed according to different methods to ensure that the dictionary always remains valid during the compression process and to avoid dictionary overflow.

[0093] As an optional implementation manner, the adaptive data compression method further includes:

[0094] Step S5: Decompress the optimal compressed codeword sequence using a corresponding data decompression algorithm to obtain decompressed data.

[0095] Step S6: Compare the decompressed data with the data to be compressed to see if they are consistent.

[0096] Step S7: If yes, the optimal compression codeword sequence is used as the final compression result of the data to be compressed.

[0097] Step S,8, if not, then update the model parameters of the adaptive compression algorithm determination model, and use the updated adaptive compression algorithm determination model to determine the target data compression algorithm.

[0098] Specifically, the corresponding LZW decompression algorithm is used to restore the original data (i.e., the data to be compressed), ensuring the data integrity during the decompression process and avoiding inconsistencies between the decompressed data and the original data.

[0099] Beneficial effects:

[0100] By establishing an adaptive compression algorithm determination model to analyze the characteristics of the data to be compressed in real time and adaptively select the optimal compression algorithm, this not only effectively improves compression efficiency but also adapts to a variety of data types, demonstrating excellent universality and compression performance. Compared to traditional compression technologies, this approach offers significant advantages when processing complex data, particularly in large-scale data storage and transmission, and holds broad application prospects.

[0101] Based on the same inventive concept, embodiments of the present application also provide an adaptive data compression system for implementing the aforementioned adaptive data compression method. The implementation solution provided by this system is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more of the adaptive data compression system embodiments provided below can be found in the above-described limitations on the adaptive data compression method and will not be further elaborated here.

[0102] In an exemplary embodiment, an adaptive data compression system is provided, comprising:

[0103] The first calculation unit is used to calculate the Shannon entropy value and repetition rate of the data to be compressed; the repetition rate of the data to be compressed is the probability of occurrence of the most frequent character in the data to be compressed.

[0104] The second calculation unit is used to divide the data to be compressed into a first half and a second half according to the data length, and calculate the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed.

[0105] The target data compression algorithm determination unit is used to determine the target data compression algorithm based on the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, the Shannon entropy value of the second half of the data to be compressed, and the adaptive compression algorithm determination model; the adaptive compression algorithm determination model is obtained by training a feedforward fully connected neural network model using a training set, and the training set includes: the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, the Shannon entropy value of the second half of the sample data, and the categories of multiple data compression algorithms.

[0106] The optimal compression codeword sequence determining unit is used to compress the data to be compressed using a target data compression algorithm to obtain an optimal compression codeword sequence.

[0107] In an exemplary embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an adaptive data compression method.

[0108] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an adaptive data compression method is implemented.

[0109] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0110] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0112] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0113] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method, system, and core concept of this application. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of this application. In summary, the contents of this specification should not be construed as limiting this application.

Claims

1. An adaptive data compression method, characterized in that: The adaptive data compression method comprises: Calculate the Shannon entropy and repetition rate of the data to be compressed; the repetition rate of the data to be compressed is the probability of occurrence of the most frequent character in the data to be compressed; Divide the data to be compressed into a first half and a second half according to the data length, and calculate the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed; A target data compression algorithm is determined based on the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, the Shannon entropy value of the second half of the data to be compressed, and an adaptive compression algorithm determination model; the adaptive compression algorithm determination model is obtained by training a feedforward fully connected neural network model using a training set, and the training set includes: the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, the Shannon entropy value of the second half of the sample data, and categories of multiple data compression algorithms; Using the target data compression algorithm to compress the data to be compressed, and obtaining the optimal compression codeword sequence; The adaptive data compression method further comprises: Decompress the optimal compressed codeword sequence using a corresponding data decompression algorithm to obtain decompressed data; Compare the decompressed data with the data to be compressed to see if they are consistent; If yes, the optimal compression codeword sequence is used as the final compression result of the data to be compressed; If not, the model parameters of the adaptive compression algorithm determination model are updated, and the updated adaptive compression algorithm determination model is used to determine the target data compression algorithm.

2. The adaptive data compression method according to claim 1, wherein: Calculate the Shannon entropy and repetition rate of the data to be compressed, including: Calculate the number of occurrences of each character in the data to be compressed; Based on the number of occurrences of each character and the length of the data to be compressed, the occurrence probability of the most frequent character in the data to be compressed is determined, and the occurrence probability of the most frequent character is used as the repetition rate of the data to be compressed; The Shannon entropy value of the data to be compressed is calculated based on the occurrence probability of each character in the data to be compressed.

3. The adaptive data compression method according to claim 1, wherein: Based on the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, the Shannon entropy value of the second half of the data to be compressed, and the adaptive compression algorithm determination model, a target data compression algorithm is determined, specifically including: Inputting the Shannon entropy value of the data to be compressed, the repetition rate of the data to be compressed, the Shannon entropy value of the first half of the data to be compressed, and the Shannon entropy value of the second half of the data to be compressed into the adaptive compression algorithm determination model, and outputting the optimal category; The data compression algorithm corresponding to each optimal category is determined as the target data compression algorithm.

4. The adaptive data compression method according to claim 3, wherein: There are one or more optimal categories, and one optimal category corresponds to one target data compression algorithm.

5. The adaptive data compression method according to claim 4, wherein: The target data compression algorithm is used to compress the data to obtain the optimal compression codeword sequence, which specifically includes: When the target data compression algorithm is one, the target data compression algorithm is used to compress the data to be compressed to obtain the optimal compression codeword sequence; When there are multiple target data compression algorithms, the data to be compressed are compressed using the multiple target data compression algorithms to obtain compression codeword sequences corresponding to the multiple target data compression algorithms; Determine the compression ratio of the compression codeword sequence corresponding to each target data compression algorithm; The compression codeword sequence corresponding to the target data compression algorithm with the highest compression rate is determined as the optimal compression codeword sequence.

6. The adaptive data compression method according to claim 1, wherein: The adaptive compression algorithm determines the model training process, which includes: Construct a training set; Build a feedforward fully connected neural network model; Taking the Shannon entropy value of sample data, the repetition rate of sample data, the Shannon entropy value of the first half of the sample data and the Shannon entropy value of the second half of the sample data as input, and the data compression algorithm as output, the feedforward fully connected neural network model is iteratively trained until the number of iterations reaches the maximum value or the loss function reaches the minimum value, and the iterative training is stopped to obtain the adaptive compression algorithm determination model.

7. The adaptive data compression method according to claim 1, wherein: The data compression algorithms include: standard LZW compression algorithm, reset LZW compression algorithm, block LZW compression algorithm, sliding window LZW compression algorithm and threshold-based dictionary reset LZW compression algorithm.

8. An adaptive data compression system, characterized in that The adaptive data compression system is used to implement the adaptive data compression method according to any one of claims 1 to 7, and the adaptive data compression system includes: The first calculation unit is used to calculate the Shannon entropy value and repetition rate of the data to be compressed; the repetition rate of the data to be compressed is the probability of occurrence of the most frequent character in the data to be compressed; A second calculation unit is used to divide the data to be compressed into a first half and a second half according to the data length, and calculate the Shannon entropy value of the first half of the data to be compressed and the Shannon entropy value of the second half of the data to be compressed; a target data compression algorithm determination unit, configured to determine a target data compression algorithm based on a Shannon entropy value of the data to be compressed, a repetition rate of the data to be compressed, a Shannon entropy value of the first half of the data to be compressed, a Shannon entropy value of the second half of the data to be compressed, and an adaptive compression algorithm determination model; the adaptive compression algorithm determination model is obtained by training a feedforward fully connected neural network model using a training set, the training set including: the Shannon entropy value of the sample data, the repetition rate of the sample data, the Shannon entropy value of the first half of the sample data, the Shannon entropy value of the second half of the sample data, and categories of multiple data compression algorithms; The optimal compression codeword sequence determining unit is used to compress the data to be compressed using a target data compression algorithm to obtain an optimal compression codeword sequence.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the adaptive data compression method according to any one of claims 1 to 7.

Images