Data compression method, device and computer storage medium

Abstract

This application provides a data compression method, apparatus, and computer storage medium. The method includes: acquiring a target file containing m characters to be compressed; classifying the m characters into a first frequency level and various second frequency levels below the first frequency level based on their frequency of occurrence in the target file, with each second frequency level corresponding to a replacement character; replacing the characters belonging to the second frequency level in the target file with their corresponding replacement characters to obtain a first file, and then compressing it to determine a first compressed file; for each second frequency level, sequentially assembling the characters belonging to the second frequency level in the target file into a second compressed file and then compressing it to determine a second compressed file; and determining the compressed file of the target file based on the first compressed file, each second compressed file, the replacement characters of each second frequency level, and the m characters. Thus, by compressing characters with lower frequency of occurrence as a whole, the overall frequency of occurrence is increased, thereby reducing the compression ratio.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data compression method, apparatus and computer storage medium. Background Technology

[0002] Reducing storage space requires file compression. Currently, file compression can be achieved using probabilistic prediction models and encoders. A probabilistic prediction model takes one or more characters preceding the character to be predicted as input and outputs a probability distribution of the character to be predicted. This probability distribution is then input into the encoder to obtain the encoded result of the character to be predicted.

[0003] However, considering that different files to be compressed have different characteristics, in order to ensure the accuracy of predictions for different files, the probabilistic prediction model needs to be trained and predicted simultaneously, resulting in low compression efficiency.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] This application provides a data compression method, apparatus, and computer storage medium that compresses characters with low frequency of occurrence as a whole, thereby increasing the frequency of occurrence of characters with low frequency of occurrence and minimizing the compression ratio.

[0006] In a first aspect, embodiments of this application provide a data compression method, comprising: obtaining a target file to be compressed; wherein the target file includes m characters; determining the occurrence frequency of each of the m characters in the target file; classifying the m characters into at least two frequency levels based on their respective occurrence frequencies; wherein the at least two frequency levels are divided into a first frequency level and various second frequency levels below the first frequency level, and each second frequency level corresponds to a replacement character; replacing the characters belonging to the second frequency level in the target file with the corresponding replacement characters to obtain a first file corresponding to the first frequency level; performing a first compression on the first file to determine a first compressed file; sequentially assembling the characters belonging to the second frequency level in the target file into a second file corresponding to the second frequency level; performing a second compression on the second files corresponding to each second frequency level to determine a second compressed file; and determining a compressed file of the target file based on the first compressed file, each second compressed file, and the replacement characters of each second frequency level.

[0007] In this method, characters with low frequency of occurrence are grouped together for compression, thereby increasing the overall frequency of these characters and minimizing the compression ratio. Furthermore, the files containing high-frequency characters differ in size from those containing low-frequency characters; the larger files of high-frequency characters achieve a higher compression ratio, while the smaller files of low-frequency characters achieve a lower compression ratio. Therefore, the combined compressed file containing both high-frequency and low-frequency characters will have a lower overall compression ratio.

[0008] In one possible implementation, the first compression of the first file includes: compressing the first file based on a first probability prediction model and a first encoder; wherein the first probability prediction model takes one character or multiple consecutive characters in the first file as the first model input and outputs a first probability distribution of p characters in the first file, and the first encoder encodes the first probability distribution to obtain the encoding result of the next character of the first model input in the first file.

[0009] In this scheme, a probabilistic prediction model and encoder are used to achieve compression. The probability of predicting the real characters is indeed relatively high. Since the higher the probability of the character, the shorter the code, the compression rate can be reduced as much as possible.

[0010] In one example, the electronic device includes a first graphics processor that runs a first probabilistic prediction model and a first encoder to compress a first file.

[0011] In this solution, a graphics processor is used to run the probabilistic prediction model and encoder, which can speed up the compression time.

[0012] In one example, the model parameters of the first probability prediction model change during the first compression of the first file.

[0013] In this scheme, an adaptive probability prediction model is used, which makes the probability of predicting real characters higher. Since the higher the probability of a character, the shorter the code, the compression rate can be reduced as much as possible.

[0014] In one example, the first probability prediction model is trained based on the first file.

[0015] In this scheme, a probabilistic prediction model is trained based on the first file, which makes the probability of the predicted real characters relatively high. Since the higher the probability of the character, the shorter the encoding, the compression rate can be reduced as much as possible.

[0016] In one possible implementation, the second file is subjected to a second compression, including:

[0017] The second file is compressed based on the second probability prediction model and the second encoder. The second probability prediction model takes one character or multiple consecutive characters in the second file as input and outputs a second probability distribution of q characters in the second file. The second encoder encodes the second probability distribution to obtain the encoding result of the next character input by the second model in the second file.

[0018] In this scheme, a probabilistic prediction model and encoder are used to achieve compression. The probability of predicting the real characters is indeed relatively high. Since the higher the probability of the character, the shorter the code, the compression rate can be reduced as much as possible.

[0019] In one example, based on the compression of the first file using the first probability prediction model, the number of model parameters of the second probability prediction model is less than that of the first probability prediction model.

[0020] In this scheme, since the number of characters with low frequency of occurrence is small, a probability prediction model with fewer model parameters can be used to speed up the compression time while ensuring prediction accuracy.

[0021] In one example, the electronic device includes a second graphics processor that runs a second probabilistic prediction model and a second encoder to compress the second file.

[0022] In this solution, a graphics processor is used to run the probabilistic prediction model and encoder, which can speed up the compression time.

[0023] In one example, the model parameters of the second probabilistic prediction model change during the second compression of the second file.

[0024] In this scheme, an adaptive probability prediction model is used, which makes the probability of predicting real characters higher. Since the higher the probability of a character, the shorter the code, the compression rate can be reduced as much as possible.

[0025] In one example, the second probability prediction model was trained based on the second file.

[0026] In this scheme, a probabilistic prediction model is trained based on the second file, which makes the probability of the predicted real characters relatively high. Since the higher the probability of the character, the shorter the encoding, the compression rate can be reduced as much as possible.

[0027] In one possible implementation, the second file is subjected to a second compression, including:

[0028] The second file is compressed a second time based on the frequency of occurrence of different characters in the third encoder and the second file.

[0029] In this solution, considering that the second file is usually small, we directly use encoding to compress it, thereby speeding up the compression time.

[0030] In one example, the electronic device includes a central processing unit that runs a third encoder to perform a second compression on the second file.

[0031] In this solution, encoding is performed on the central processing unit, eliminating the need for an additional graphics processor for compression, which reduces the cost of using the algorithm.

[0032] Secondly, embodiments of the present invention provide a data compression apparatus comprising several modules, each module being used to execute various steps in the data compression method provided in the first aspect of the present invention. The division of modules is not limited herein. For the specific functions performed by each module of this data compression apparatus and the beneficial effects achieved, please refer to the functions of each step in the data compression method provided in the first aspect of the present invention; further details will not be repeated here.

[0033] For example, the data compression device includes:

[0034] The file acquisition module is used to acquire the target file to be compressed; the target file consists of m characters.

[0035] The character segmentation module is used to determine the frequency of each of the m characters in the target file; based on the frequency of each of the m characters, the m characters are classified into characters with at least two frequency levels; wherein, the at least two frequency levels are divided into a first frequency level and various second frequency levels below the first frequency level, and each second frequency level corresponds to a replacement character;

[0036] The file classification module is used to replace characters belonging to the second frequency level in the target file with corresponding replacement characters to obtain the first file corresponding to the first frequency level; and to assemble the characters belonging to the second frequency level in the target file into the second file corresponding to the second frequency level in sequence.

[0037] The first compression module is used to perform a first compression on the first file and determine the first compressed file;

[0038] The second compression module is used to perform a second compression on the second file corresponding to each second frequency level, and determine the second compressed file.

[0039] The third compression module is used to determine the compressed file of the target file based on the first compressed file, each of the second compressed files, and the replacement characters of each of the second frequency levels.

[0040] The beneficial effects of this solution are described above and will not be repeated here.

[0041] According to a feasible implementation, a first compression module is used to perform a first compression on a first file based on a first probability prediction model and a first encoder; wherein, the first probability prediction model takes one character or multiple consecutive characters in the first file as the first model input and outputs a first probability distribution of p characters in the first file, and the first encoder encodes the first probability distribution to obtain the encoding result of the next character of the first model input in the first file.

[0042] In one example, the first compression module is the first graphics processor.

[0043] In one example, the model parameters of the first probability prediction model change during the first compression of the first file.

[0044] In one example, the first probability prediction model is trained based on the first file.

[0045] According to a feasible implementation, a second compression module is used to compress a second file based on a second probability prediction model and a second encoder; wherein, the second probability prediction model takes one character or multiple consecutive characters in the second file as the second model input and outputs a second probability distribution of q characters in the second file; the second encoder encodes the second probability distribution to obtain the encoding result of the next character of the second model input in the second file.

[0046] In one example, based on the compression of the first file using the first probability prediction model, the number of model parameters of the second probability prediction model is less than that of the first probability prediction model.

[0047] In one example, the second compression module is a second graphics processor.

[0048] In one example, the model parameters of the second probabilistic prediction model change during the first compression of the second file.

[0049] In one example, the second probability prediction model was trained based on the second file.

[0050] According to one feasible implementation, a second compression module is used to perform a second compression on the second file based on the frequency of occurrence of different characters in the third encoder and the second file.

[0051] In one example, the second compression module is the central processing unit.

[0052] The beneficial effects are described above and will not be repeated here.

[0053] Thirdly, embodiments of this application provide a data compression apparatus, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method provided in the first aspect.

[0054] Fourthly, embodiments of this application provide a data compression apparatus, characterized in that the apparatus executes computer program instructions to perform the method provided in the first aspect. Exemplarily, the apparatus may be a chip or a processor.

[0055] In one example, the device may include a processor that may be coupled to memory, read instructions from the memory, and execute the methods provided in the first aspect according to those instructions. The memory may be integrated into the chip or processor, or it may be independent of the chip or processor. For example, the processor may include a central processing unit (CPU) and may also include several graphics processing units (GPUs).

[0056] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect.

[0057] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. Attached Figure Description

[0058] Figure 1 This is a schematic flowchart of a data compression method provided in an embodiment of this application;

[0059] Figure 2 This is a schematic diagram of a specific example of a data compression method provided in the embodiments of this application. Figure 1 ;

[0060] Figure 3 yes Figure 2 A schematic diagram illustrating a specific example of the decompression method of the aforementioned compression method;

[0061] Figure 4 This is a schematic diagram of a specific example of a data compression method provided in the embodiments of this application. Figure 2 ;

[0062] Figure 5 yes Figure 4 A schematic diagram illustrating a specific example of the decompression method of the aforementioned compression method;

[0063] Figure 6 This is a schematic diagram of a specific example of a data compression method provided in the embodiments of this application. Figure 3 ;

[0064] Figure 7 yes Figure 6 A schematic diagram illustrating a specific example of the decompression method of the aforementioned compression method;

[0065] Figure 8 This is a schematic diagram of a specific example of a data compression method provided in the embodiments of this application. Figure 4 ;

[0066] Figure 9 yes Figure 8 A schematic diagram illustrating a specific example of the decompression method of the aforementioned compression method;

[0067] Figure 10 This is a schematic diagram of the structure of the data compression device provided in the embodiments of this application;

[0068] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0070] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0071] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0072] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0073] To reduce storage space, it is usually necessary to compress the target file to be compressed.

[0074] In related technologies, if the target file to be compressed is a sequence with related context and the symbols in the sequence are not independent of each other, in practical applications, the design of the compressor is divided into two parts: one is a probability prediction model, and the other is an encoder that compresses according to the probability distribution.

[0075] The probabilistic prediction model is used to predict the probability distribution of the next symbol in a given context of multiple consecutive symbols in a target file to be compressed. This probability distribution is formed by the probability of each possible symbol. For example, assuming the given context is "We are researche", there are 256 possible symbols, each with a probability of occurrence. For instance, the probability of symbol "a" might be 1%, and the probability of symbol "b" might be 1.2%. If, given a given context of a certain length, the probability of the symbol to be compressed is the highest among the 256 symbols, then the prediction is considered accurate. For example, for the context "We are", the next symbol to be compressed is "r". If, in the probability distribution, "r" has the highest probability among the 256 possible symbols, then the prediction is accurate. That is, P(r|We_are_researche) = max(P(i|We_are_researche)), where i is any symbol. Generally, the longer the context, the more accurate the symbol prediction. For example, the context "We are research" will predict the next symbol "e" better than the context "rch". Generally, higher prediction accuracy corresponds to higher compression ratio.

[0076] During decompression, the next symbol can be accurately predicted based on any k contexts (i.e., the number of symbols contained in the context is k). Therefore, for any length sequence, only the first k symbols need to be recorded. That is, first use symbols 1 to k as context to obtain the (k+1)th symbol, then use symbols 2 to k+1 as context to obtain the (k+2)th symbol, and so on, thus restoring the target file to be compressed.

[0077] For encoders, the encoding method can be arithmetic coding or Huffman coding. Arithmetic coding represents a sequence of symbols as an interval between 0s and 1s, and uses a floating-point number within that interval, then converts the fractional number into a binary number. The longer the symbol sequence, the smaller the corresponding interval, and the more binary bits are used to represent that interval. Huffman coding, also known as Huffman encoding, is an encoding method that constructs the shortest codeword on average based entirely on the probability of character occurrence.

[0078] For probabilistic prediction models, in related technologies, probabilistic prediction models can be AI (Artificial Intelligence) models, especially deep learning network-based AI models, capable of data classification and prediction. Common AI models are primarily RNN (Recurrent Neural Network) models. An RNN is a neural network specifically designed for sequential data; it can be trained to identify dependencies in long sequences and can be used as a probabilistic predictor in compression algorithms. Specifically, for the symbol "a" to be compressed... i "The context is of length k, and the sequence {a} of length k is taken." i-k ,a i-k+1 ,…,a i-1 As features, all possible symbols form an alphabet of size d as the prediction space. The entire target file to be compressed is used for training, and the loss is defined as the cross-entropy of the classification error, which is transformed into a classification problem.

[0079] After estimating the model parameters using the backpropagation algorithm, they are combined with an entropy encoder. Specifically, the initial k symbols are saved. For the (k+1)th symbol, the first to k symbols are used as input to the RNN to obtain the probability distribution of the alphabet. This probability distribution is then fed into the encoder for the first encoding, resulting in the symbol "a". k+1 The corresponding encoding result; then, the 2nd to k+1th symbols are used as input to the RNN to obtain the probability distribution of the alphabet, and this probability distribution is fed into the entropy encoder to obtain the symbol "a". k+2 The corresponding encoding interval is repeated in a loop to compress the data and obtain a compressed file.

[0080] The decompression process is symmetrical. The initial k characters are uncompressed and can be directly read as input to the RNN to obtain a probability distribution. The probability distribution undergoes a first encoding process to obtain the compression results corresponding to each symbol, and the symbol "a" is decompressed.k+1 Then, the 2nd to k+1th symbols are used as inputs to the RNN, and so on, to achieve data decompression.

[0081] One problem with this approach is that AI-based probabilistic prediction models essentially achieve probabilistic prediction through network parameters. There are three possible solutions for storing these model parameters:

[0082] The first approach involves an overfitted probabilistic prediction model: This model is trained to overfit as much as possible to ensure maximum prediction accuracy. In other words, the trained model, after inputting k consecutive symbols from the target file to be compressed, predicts a probability distribution where the probability of the next symbol following those k consecutive symbols is maximized. Since the network structure, parameter size, and storage space are fixed, the network parameters are negligible for very large files. Under this approach, training the probabilistic prediction model is very slow because it requires repeated training to achieve overfitting. Conversely, the decompression process is relatively fast because the network parameters are stored, and only the forward propagation process is involved.

[0083] The second approach is the probabilistic prediction model using transfer learning: A large probabilistic prediction model is trained on a large training set (such as an encyclopedia), and then stored as the size of the compression software itself. When compressing new files, the probability distribution can be obtained simply through forward propagation, without involving model training. The advantage of this transfer learning approach is that both compression and decompression only require forward propagation, resulting in high speed. The disadvantage is that the compression ratio depends on the similarity of the distribution between the target file and the training file.

[0084] Thirdly, there is the adaptive probabilistic prediction model: The network parameters are initialized. Each input of a context of length k yields a probability distribution, even if this distribution is inaccurate. This probability distribution is fed into the entropy encoder for compression, and then the loss function is calculated. Backpropagation is used to adjust the model parameters of the probabilistic prediction model. In this approach, the estimation of the model parameters and file compression occur simultaneously. The probabilistic prediction model gradually evolves from a coarse structure to a fine predictor, and the compression ratio gradually increases and stabilizes. In practical applications, this method eliminates the need to record model parameters; only the random number seed needs to be recorded. The disadvantage of this method is that both compression and decompression involve model training, which is slow. Furthermore, the initial network is a coarse probabilistic predictor with a low initial compression ratio, which gradually increases as compression progresses. If the file size is small, compression may terminate before the parameters are fully adjusted. For cases where file size tends towards infinity and file distribution is stable, adaptive and overfitting methods are equivalent.

[0085] AI-based probabilistic prediction has the following two problems.

[0086] Problem 1: Training is very slow in the compression process for overfitting and self-adaptive probability predictions.

[0087] Question 2: For characters that appear less frequently, the compression ratio will be increased.

[0088] To address the problem of AI-based probabilistic prediction, this invention provides the following data compression method.

[0089] First, obtain the target file to be compressed; the target file includes m characters; then, determine the frequency of each of the m characters in the target file; then, based on the frequency of each of the m characters, classify the m characters into characters of at least two frequency levels; the at least two frequency levels are divided into a first frequency level and various second frequency levels below the first frequency level, each second frequency level corresponding to a replacement character; then, replace the characters belonging to the second frequency level in the target file with the corresponding replacement characters to obtain the first file corresponding to the first frequency level; then, perform a first compression on the first file to determine the first compressed file; then, arrange the characters belonging to the second frequency level in the target file in order to form the second compressed files corresponding to the second frequency level; then, perform a second compression on the second files corresponding to each second frequency level to determine the second compressed file; finally, based on the first compressed file and each second compressed file, determine the compressed file of the target file.

[0090] In summary, this embodiment of the invention compresses all characters with low frequency of occurrence as a single category, thereby increasing the overall frequency of occurrence of these characters and minimizing the compression ratio. Furthermore, the files containing high-frequency characters and low-frequency characters differ in size; the files containing high-frequency characters are larger and therefore have a higher compression ratio, while the files containing low-frequency characters are smaller and therefore have a lower compression ratio. Therefore, the combined compressed file containing both high-frequency and low-frequency characters will have a relatively low compression ratio.

[0091] Figure 1 This is a schematic flowchart of the data compression method provided in the embodiments of this application. This method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities, such as an electronic device. The data compression method provided in the embodiments of this invention will be described below in conjunction with an electronic device.

[0092] like Figure 1 As shown, the data compression method provided in this application embodiment includes at least the following steps:

[0093] Step 101: Obtain the target file to be compressed; the target file consists of m characters.

[0094] Here, the target file consists of m characters, and these m characters appear repeatedly in the target file.

[0095] Step 102: Determine the frequency of each of the m characters in the target file.

[0096] According to a feasible implementation method, the frequency of occurrence of multiple characters in the target file to be compressed is statistically analyzed to obtain the frequency of occurrence of each character.

[0097] According to one feasible implementation, if the characters in the target file to be compressed are encoded in ASCII, then m = 256.

[0098] According to one feasible implementation, the frequency of occurrence indicates the number of times m characters appear in the target file.

[0099] In one example, the frequency of occurrence is the number of times a character m appears in the target file.

[0100] In one example, the frequency is the ratio of the number of times m characters appear in the target file to m.

[0101] Step 103: Based on the occurrence frequency of each of the m characters, classify the m characters into characters with at least two frequency levels; wherein, the at least two frequency levels are divided into a first frequency level and each second frequency level below the first frequency level, and each second frequency level corresponds to a replacement character.

[0102] Here, at least two frequency levels can have n (greater than or equal to 2) frequency levels. In other words, there are n character categories, and different types of characters have different frequency levels. The frequency intervals to which the n frequency levels belong do not overlap, and they can cover the occurrence frequencies of the m different characters in the target file to be compressed.

[0103] According to a feasible implementation method, based on the occurrence frequency of each of the m characters, the m characters are classified into characters with at least two frequency levels.

[0104] In one example, the frequency level information is divided into n frequency intervals, and each frequency interval is a frequency level. Then, based on the n frequency intervals and the occurrence frequency of each of the multiple characters, characters whose occurrence frequency is in the same frequency interval are classified as belonging to the same frequency level. This determines the multiple characters corresponding to each of the n frequency intervals, resulting in the characters of the n frequency levels.

[0105] In one example, the frequency level classification information consists of n-1 cutoff values.

[0106] Optionally, the interval [0,1] can be divided based on n-1 cutoff values ​​to obtain n frequency intervals. Each frequency interval serves as a frequency level, and characters at n frequency levels can be obtained in the aforementioned manner. Correspondingly, the frequency of a character's occurrence is greater than or equal to 0 and less than 1.

[0107] Optionally, based on the occurrence frequency of each character, the characters are sorted in descending order of occurrence frequency to determine a sorted character table; based on the pre-set n-1 cutoff values ​​and the occurrence frequency of each character, the sorted character table is divided to obtain n frequency levels of characters.

[0108] For example, if n=2, then a pre-set cutoff value and the frequency of occurrence of each character can be used to determine the first and second categories of characters. Characters in the first category have a frequency greater than or equal to the cutoff value, while characters in the second category have a frequency less than the cutoff value. It's important to note that characters with frequencies above the cutoff value can be considered high-frequency characters, and those below can be considered low-frequency characters. Specifically, based on the pre-set cutoff value and the frequency of occurrence of each character, the dividing point I in the sorted character table is determined, i.e., the I-th character symbol. The frequency of occurrence of each of the first to the I-th symbols in the sorted character table is greater than or equal to the cutoff value.

[0109] Step 104: Replace the characters in the target file that belong to the second frequency level with the corresponding replacement characters to obtain the first file corresponding to the first frequency level.

[0110] For ease of description and distinction, the highest frequency level among the n frequency levels is designated as the first frequency level, which can be referred to as first frequency level A. All frequency levels other than first frequency level A are designated as second frequency levels, which can be referred to as second frequency levels B. Each second frequency level B has a corresponding replacement character, and the replacement characters differ between different second frequency levels B. Specifically, the replacement characters are characters other than those belonging to first frequency level A, and the replacement characters differ between different second frequency levels B.

[0111] Specifically, for the first frequency level A, the characters belonging to the second frequency level B in the target file to be compressed are replaced with the corresponding replacement characters of the second frequency level B, and the resulting file is used as the first file corresponding to the first frequency level A. For ease of description and distinction, the first file is referred to as the first file Ac.

[0112] Step 105: Perform a first compression on the first file to determine the first compressed file.

[0113] According to a feasible implementation, the first file Ac can be compressed using a first probabilistic prediction model (referred to as the first probabilistic prediction model 1 for ease of description and distinction) and a first encoder (referred to as the first encoder 1 for ease of description and distinction) to obtain the first compressed file corresponding to the first file Ac (referred to as the first compressed file Ar for ease of description and distinction). For ease of description, assume that the first file Ac contains a total of p different characters, that is, the number of characters under the first frequency level A and the number of all replacement characters under the second frequency level B are p.

[0114] In one example, the model parameters of the first probability prediction model 1 will change during the first compression process.

[0115] For example, the first probability prediction model 1 adjusts its parameters after each prediction. The specific compression method is as follows:

[0116] A1. Input the first 1 to k symbols from the beginning of the first file Ac into the first probability prediction model 1 (model parameter mp0) to obtain the probability distribution P10 of m1 characters. Input the probability distribution P10 into the first encoder 1 for encoding to obtain the encoding results of p characters. Select the (k+1)th symbol "a" from the first file Ac. k+1 The corresponding encoding result Ck+1 is based on the probability distribution P10 and the (k+1)th symbol "a". k+1 "Calculate the error E0k+1, adjust the model parameter mp0 of the first probability prediction model 1 Model0 based on the error, and obtain the model parameter mp1."

[0117] A2. Input the 2nd to k+1th symbols from the first file Ac into the first probability prediction model 1 (model parameter mp1) to obtain the probability distribution P20 of p characters. Input the probability distribution P20 into the first encoder 1 coder for encoding to obtain the symbol "a". k+2 The corresponding encoding result is Ck+2; then, based on the probability distribution P20 and the (k+2)th symbol "a" k+2 "Calculate the error E0k+2, input the 1st to kth symbols into the first probability prediction model 1 (model parameter is mp1) to obtain the probability distribution P11, and based on the probability distribution P11 and the (k+1)th symbol "a k+1"Calculate the error E1k+1. Based on the error E1k+1 and the error E0k+2, adjust the model parameter mp1 of the first probability prediction model 1 to obtain the model parameter mp2."

[0118] A3. Input the 3rd to k+2th symbols from the first file Ac into the first probability prediction model 1 (model parameter mp2) to obtain the probability distribution P3 of p characters. Input the probability distribution P3 into the first encoder 1 coder for encoding to obtain the symbol "a". k+3 The corresponding encoding result is Ck+3; then, based on the probability distribution P3 and the (k+3)th symbol "a" k+3 "Calculate the error E0k+3, input the 1st to kth symbols into the first probability prediction model 1 (model parameter is mp2) to obtain the probability distribution P12, and based on the probability distribution P12 and the (k+1)th symbol "a k+1 "Calculate the error E2k+1, input the 2nd to k+1st symbols into the first probability prediction model 1 (Model2) to obtain the probability distribution P21. Based on the probability distribution P21 and the k+2nd symbol "a..." k+2 "Calculate the error E1k+2. Based on the errors E2k+1, E2k+2, and E0k+3, adjust the model parameter mp2 of the first probability prediction model 1 to obtain the model parameter mp3."

[0119] The process is repeated to compress the data and obtain the first compressed file, Ar.

[0120] For example, the first probability prediction model 1 adjusts the model parameters in batches. Assuming the batch size is X, the specific compression method is as follows:

[0121] B1. Input the first 1 to k symbols from the beginning of the first file Ac into the first probability prediction model 1 (model parameter mp0) to obtain the probability distribution P10 of p characters. Input the probability distribution P10 into the first encoder 1 to encode the m characters to obtain the encoding results of each character. Select the (k+1)th symbol "a" from the first file Ac. k+1 The corresponding encoding result is Ck+1. Then, based on the probability distribution P10 and the (k+1)th symbol "a",... k+1 "Calculation error E0k+1."

[0122] B2. Input the 2nd to k+1th symbols from the first file Ac into the first probability prediction model 1 (model parameter mp0) to obtain the probability distribution P20 of p characters. Input the probability distribution P20 into the first encoder 1 coder for encoding to obtain the symbol "a". k+2 The corresponding encoding result is Ck+2. Then, based on the probability distribution P20 and the (k+2)th symbol "a",... k+2 "Calculation error E0k+2."

[0123] ...

[0124] BX+1: Input the (X+1)th to (k+X)th symbols in the first file Ac into the first probability prediction model 1 (model parameter mp0) to obtain the probability distribution PX0 of p characters. Input the probability distribution PX0 into the first encoder 1 for encoding to obtain the symbol "a". k+x+1 The corresponding encoding result is Ck+X+1. Then, based on the probability distribution PX0 and the (k+X+1)th symbol "a",... k+X+1 "Calculate the error E0k+X+1. Then, based on the errors E0k+1, ..., E0k+X+1, adjust the model parameter mp0 of the first probability prediction model 1 to obtain the model parameter mp1."

[0125] BX+2. Input the symbols from the (X+2)th to the (k+X+1)th character in the first file Ac into the first probability prediction model 1 (model parameter is mp1) to obtain the probability distribution P(X+1)0 of p characters. Input the probability distribution P(X+1)0 into the first encoder 1 coder for encoding to obtain the symbol "a". k+x+2 The corresponding encoding result is Ck+X+2. Then, based on the probability distribution P(X+1)0 and the k+X+2th symbol "a",... k+X+2 "Calculate the error E0k+X+2."

[0126] ...

[0127] B2X+1: Input the (X+1)th to (k+2)Xth symbols from the first file Ac into the first probability prediction model 1 (model parameter mp1) to obtain the probability distribution P(2X+1)0 of p characters. Input the probability distribution P(2X+1)0 into the first encoder 1 coder for encoding to obtain the symbol "a". k+2x+1 The corresponding encoding result is Ck+2X+1. Then, based on the probability distribution P(2X+1)0 and the (k+2X+1)th symbol "a",... k+2X+2"Calculate the error E0k+2X+2. Then, the first to k symbols are input into the first probability prediction model 1, Model1, to obtain the probability distribution P11. Based on the probability distribution P11 and the (k+1)th symbol "a..." k+1 "Calculate the error E1k+1, and so on, to obtain the errors E1k+2, ..., E1k+X+1. Then, based on the errors E1k+1, ..., E1k+X+1, E0k+X+2, ..., E0k+2X+2, adjust the model parameter mp1 of the first probability prediction model 1 to obtain the model parameter mp2."

[0128] The process is repeated to compress the data and obtain the first compressed file, Ar.

[0129] In one example, the first probability prediction model 1 is a model trained on the first file Ac. Assuming the first file Ac contains k+y characters, the specific compression method is as follows:

[0130] C1. Input the first to k symbols from the beginning of the first file Ac into the first probability prediction model 1 (model parameter is mp0) to obtain the probability distribution P10 of p characters. Based on the probability distribution P10 and the (k+1)th symbol "a k+1 "Calculation error E0k+1."

[0131] C2. Input the 2nd to k+1th symbols from the first file Ac into the first probability prediction model 1 (model parameter is mp0) to obtain the probability distribution P20 of p characters. Based on the probability distribution P20 and the k+2th symbol "a", k+2 "Calculation error E0k+2."

[0132] ...

[0133] Cy, input the y-th to k+y-1th symbols from the first file Ac into the first probability prediction model 1 (model parameter mp0), to obtain the probability distribution Py0 of p characters. Based on the probability distribution Py0 and the k+y-th symbol "a k+y "Calculate the error E0k+y."

[0134] Then, based on the calculation errors E0k+1, ..., E0k+y, the first probability prediction model 1 (with model parameter mp0) can be adjusted to obtain model parameter mp1. Then, the model parameter mp0 of the first probability prediction model 1 in C1 to Cy can be replaced with mp1. Based on the obtained errors E1k+1, ..., E1k+y, the model parameter mp1 can be adjusted to obtain model parameter mp2. This process is repeated to obtain the model parameter with better performance, which is assumed to be mpi.

[0135] Furthermore, the compression process is as follows:

[0136] D1. Input the first to k symbols from the beginning of the first file Ac into the first probability prediction model 1 (model parameter is mpi) to obtain the probability distribution P10 of p characters. Input the probability distribution P10 into the first encoder 1 to encode the p characters and obtain the encoding results corresponding to each character. Select the (k+1)th symbol "a" from the first file Ac. k+1 The corresponding encoding result is Ck+1.

[0137] D2. Input the 2nd to k+1th symbols from the first file Ac into the first probability prediction model 1 (model parameter is mpi) to obtain the probability distribution P20 of p characters. Input the probability distribution P20 into the first encoder 1 coder for encoding to obtain the symbol "a". k+2 The corresponding encoding result is Ck+2.

[0138] D3. Input the 3rd to k+2th symbols from the first file Ac into the first probability prediction model 1 (model parameter is mpi) to obtain the probability distribution P3 of p characters. Input the probability distribution P3 into the first encoder 1 coder for encoding to obtain the symbol "a". k+3 The corresponding encoding result is Ck+3.

[0139] The process is repeated to compress the data, resulting in a first compressed file, Ar. Correspondingly, the first compressed file Ar needs to include the initial model parameters mp0 of the first probabilistic prediction model 1, ensuring that the initial model parameters mp0 of the first probabilistic prediction model 1 are consistent with those used in the compression process. Of course, if the initial model parameters mp0 of the first probabilistic prediction model 1 are consistent between the compression and decompression processes, then it is not necessary to record the initial model parameters mp0 of the first probabilistic prediction model 1.

[0140] Furthermore, for the compression method described above, if the first encoder 1coder uses arithmetic encoding, for A2, B2, and D2, it is necessary to combine the probability distribution P20 and the (k+1)th symbol "a k+1 The corresponding encoding result Ck+1 is input into the first encoder 1coder for arithmetic encoding to obtain the symbol "a". k+2 The corresponding encoding result is Ck+2. For A3, B3, and D3, the probability distribution P3 and the symbol "a" are... k+2 The corresponding encoding result Ck+2 is input into the first encoder 1coder for arithmetic encoding to obtain the symbol "a". k+3 The corresponding encoding result is Ck+3. Correspondingly, the first compressed file Ar, which corresponds to the first file Ac, includes the compressed result of the last character in the first file Ac.

[0141] If the first encoder 1 uses Huffman coding, it can directly construct a tree based on the probability distribution to obtain the encoding results for each of the p characters. Therefore, for A2, B2, and D2, the probability distribution P20 is input into the first encoder 1 to construct the tree, resulting in the symbol "a". k+2 The corresponding encoding result is Ck+2. For A3, B3, and D3, the probability distribution P3 is input into the first encoder 1coder constructing the tree to obtain the symbol "a". k+3 The corresponding encoding result is Ck+3. Correspondingly, the first compressed file Ar corresponding to the first file Ac includes the compressed result sequence after concatenating the compressed results from the (k+1)th character to the last character in the first file Ac.

[0142] It should be noted that in practical applications, the input of the first probability prediction model can be adaptively varied. For example, the first probability prediction model can be all characters from the beginning of the first file Ac to the characters to be compressed. For example, for A2 and B2, the input of the first probability prediction model can be the first to the (k+1)th characters in the first file, and for A3 and B3, the input of the first probability prediction model can be the first to the (k+2)th characters in the first file.

[0143] Step 106: Replace the characters in the target file that belong to the second frequency level with the corresponding replacement characters to obtain the second file corresponding to the second frequency level.

[0144] According to a feasible implementation method, for each second frequency level B, the characters and order belonging to the second frequency level B in the target file to be compressed are retained, and the resulting file is used as the second file corresponding to the second frequency level B. For ease of description and distinction, the second file is called the second file Bc.

[0145] Step 107: For each second file corresponding to a second frequency level, perform a second compression on the second file to determine the second compressed file.

[0146] According to a feasible implementation, the second file Bc can be compressed using a second probabilistic prediction model (referred to as the second probabilistic prediction model 2 for ease of description and distinction) and a second encoder (referred to as the second encoder 2 for ease of description and distinction) to obtain a second compressed file corresponding to the second file Bc (referred to as the second compressed file Br for ease of description and distinction). For ease of description, assume that the second file Bc contains a total of q different characters, that is, the number of characters belonging to frequency level B among m characters is q. In practical applications, the number of characters contained in different second files Bc is generally different.

[0147] It is worth noting that each second file Bc corresponds to a second probability prediction model 2Model.

[0148] In one example, the model parameters of the second probabilistic prediction model 2 will change during the second compression process. For details, please refer to the description of the first probabilistic prediction model 1 and the first encoder 1 above. The difference is that the probability distribution of p characters is the same as the probability distribution of q characters, which will not be repeated here.

[0149] In one example, the second probabilistic prediction model 2Model is a model trained based on the second file Bc. For details, please refer to the description of the first probabilistic prediction model 1Model and the first encoder 1coder above, which will not be repeated here.

[0150] It is worth noting that in practical applications, the second compressed file Br includes the initial model parameters of the second probabilistic prediction model 2Model, thus ensuring that the initial model parameters of the second probabilistic prediction model 2Model are consistent with those used in the compression process during decompression. Of course, if the initial model parameters of the second probabilistic prediction model 2Model used in the compression and decompression processes are consistent, then it is not necessary to record the initial model parameters of the second probabilistic prediction model 2Model.

[0151] Here, the number of model parameters in the second probabilistic prediction model 2 is less than that in the first probabilistic prediction model 1. It should be noted that the compression speed of the first file Ac and the second file Bc is significantly improved due to the reduction in the number of model parameters in the probabilistic prediction models.

[0152] According to a feasible implementation, the second file Bc can be directly compressed using a third encoder (for ease of description and distinction, it can be called the third encoder 3coder) to obtain the second compressed file Br corresponding to the second file Bc. It should be noted that by using a probabilistic prediction model for the first file Ac and directly encoding the second file Bc using an encoding algorithm, the compression speed of the first file Ac and the second file Bc can be effectively improved. At the same time, the encoding method of the encoder is relatively simple, which can effectively reduce the algorithm cost.

[0153] If the third encoder 3coder uses arithmetic encoding, and the frequency of occurrence of mB characters in the second file Bc is taken as the probability distribution P, and the probability distribution P is divided into intervals to obtain the encoding results corresponding to q characters, where each encoding result is an interval; then the specific encoding process is as follows:

[0154] E1. For the first symbol in the second file Bc, determine the encoding result C1 corresponding to the first character symbol from the encoding results corresponding to each of the q characters.

[0155] E2. For the second symbol in the second file Bc, according to the interval ratio of the encoding result C0 corresponding to each of the q characters, divide the encoding result C1 corresponding to the first character symbol in the same proportion to obtain the encoding results of various characters for m2 characters, and select the encoding result C2 corresponding to the second character symbol "a2" from them.

[0156] E3. For the third symbol in the second file Bc, according to the interval ratio of the encoding result C0 corresponding to each of the q characters, divide the encoding result C2 corresponding to the second character symbol in the same proportion to obtain the encoding results of various characters for m2 characters, and select the encoding result C3 corresponding to the second character symbol "a3".

[0157] This process is repeated to compress the data, resulting in a second compressed file Br. Correspondingly, the second compressed file Br, which corresponds to the second file Bc, includes the compressed result of the last character in the second file Bc.

[0158] If the third encoder 3coder adopts Huffman coding, the third encoder 3coder can directly construct a tree based on the probability distribution P of the occurrence frequency of q characters to obtain the encoding results corresponding to each of the q characters; correspondingly, the second compressed file Br corresponding to the second file Bc includes the compressed result sequence after each character in the second file Bc is replaced with the corresponding encoding result.

[0159] It should be noted that the characters in the second file appear less frequently, resulting in a higher compression ratio, while the characters in the first file appear more frequently, resulting in a lower compression ratio. However, since the second file is much smaller than the first file, the combined compressed file of the second and first files will have a considerable compression ratio.

[0160] Step 108: Based on the first compressed file, each of the second compressed files, the replacement characters of each second frequency level, and m characters, determine the compressed file of the target file.

[0161] It should be noted that parsing the compressed files of the target file can at least obtain the replacement characters of the first compressed file Ar, each of the second compressed files Br, and each of the second frequency levels B.

[0162] According to one feasible implementation, the second compressed file Br is encoded using a third encoder 3coder. Correspondingly, the compressed file of the target file includes characters of each second frequency level B and their frequency of occurrence in the target file.

[0163] According to a feasible implementation, the first compressed file Ar uses a first probability prediction model 1 for probability prediction, and the second compressed file Br uses a second probability prediction model 2 for probability prediction. It is worth noting that the first probability prediction model 1 includes multiple character numbers arranged in a certain order and a probability value for each character number.

[0164] Optionally, assuming that the character numbers in the probability distribution output by the first probability prediction model 1 are the sorted numbers of all characters under the character encoding type, then the compressed file of the target file does not need to restore the characters under the first frequency level A; correspondingly, the compressed file of the target file needs to include the character encoding type. Here, the character encoding type can be ASCII (American Standard Code for Information Interchange), GB2312, GBK, GB18030, Unicode, or UTF-8.

[0165] In the subsequent decompression process, the sorting number of all characters under the character encoding type can be restored based on the character encoding type. Based on this, the character corresponding to each character number in the probability distribution output by the first probability prediction model 1 is obtained, thereby achieving decompression.

[0166] Optionally, assuming that the character number in the probability distribution output by the second probability prediction model 1 is the sorted number after sorting all the characters under the character encoding type, then the compressed file of the target file does not need to restore the characters under the second frequency level A; correspondingly, the compressed file of the target file needs to include the character encoding type.

[0167] In the subsequent decompression process, the sorting number of all characters under the character encoding type can be restored based on the character encoding type. Based on this, the character corresponding to each character number in the probability distribution output by the second probability prediction model 2Model can be obtained, thereby achieving decompression.

[0168] Optionally, assuming that the character numbers in the probability distribution output by the first probability prediction model 1 are not the sorted numbers of all characters under the character encoding type, the compressed file of the target file may include a first character table, which records the character numbers and the characters corresponding to the character numbers in the probability distribution output by the first probability prediction model 1. These characters consist of the characters under the first frequency level A and the replacement characters corresponding to each second frequency level B, for a total of p characters.

[0169] Optionally, assuming that the character numbers in the probability distribution output by the second probability prediction model 2 are not the sorted numbers of all characters under the character encoding type, the compressed file of the target file may include a second character table, which records the character numbers in the probability distribution output by the second probability prediction model 2 and the characters corresponding to the character numbers, and these characters are composed of characters under the second frequency level B.

[0170] The following is a specific example A. Assume that the character numbers in the probability distributions output by the first probability prediction model 1 and the second probability prediction model 2 are obtained by sorting the characters in descending order of frequency. Simultaneously, the frequency segmentation information described in step 103 is used to classify the characters.

[0171] The compressed target file includes frequency division information and a third character table (containing m characters and their frequencies, sorted from highest to lowest frequency; this sorting is merely an example, and in practice, frequencies from lowest to highest can be used). During decompression, based on the frequency division information and the frequencies of the m characters in the third character table, characters for multiple frequency levels can be reconstructed, yielding characters for the first frequency level A and each of the second frequency levels B. Combining this with the character order in the third character table, the character at each position in the probability distribution output by the first probability prediction model 1 and the second probability prediction model 2 can be obtained. It is noteworthy that replacing all characters belonging to the second frequency level B in the third character table with their corresponding replacement characters ensures that the order of the resulting character table matches the character encoding order in the probability distribution output by the first probability prediction model 1.

[0172] Below is another specific example B. Assume that the character numbers in the probability distribution output by the first probability prediction model 1 are in the order of the entire character set under the character encoding type; and the character numbers in the probability distribution output by the second probability prediction model 2 are obtained by sorting the characters according to their frequency from highest to lowest. The compressed file of the target file needs to restore the characters under the first frequency level A and the characters under the second frequency level B.

[0173] The compressed target file can include a fourth character table and a character encoding type. The fourth character table sequentially records characters at each of the second frequency levels, arranged in descending order of frequency. This sorting order is merely an example; in practice, characters can be sorted in ascending order of frequency. During decompression, the full set of characters under the character encoding type can be obtained. Characters other than those at each of the second frequency levels belong to the first frequency level A. The characters at first frequency level A and all replacement characters at second frequency level B are arranged according to the order of the full set of characters under the character encoding type. The resulting character table corresponds to the probability distribution output by the first probability prediction model 1, thus achieving decompression.

[0174] In this embodiment of the invention, characters with low frequency of occurrence are grouped together for compression, thereby increasing the frequency of occurrence of these characters and minimizing the compression ratio. Furthermore, the files containing high-frequency characters and low-frequency characters differ in size; the files containing high-frequency characters are larger and therefore have a higher compression ratio, while the files containing low-frequency characters are smaller and therefore have a lower compression ratio. Therefore, the combined compressed file containing both high-frequency and low-frequency characters will have a relatively low compression ratio.

[0175] Furthermore, the decompression method of the compression method provided in this embodiment of the invention is as follows:

[0176] Based on the target file's compressed files, obtain the first compressed file Ar, each of the second compressed files Br, and the replacement characters for each second frequency level.

[0177] Then, the first compressed file Ar is decompressed using the same compression method used for the first file Ac, for example, by compressing the first probabilistic prediction model 1Model and the first encoder 1coder, resulting in the first file Ac. It is worth noting that in practical applications, if compression using the first probabilistic prediction model 1Model and the first encoder 1coder is employed, the first probabilistic prediction model 1Model needs to be initialized based on the initial model parameters in the first compressed file Ar. This ensures that the initial model parameters of the first probabilistic prediction model 1Model are consistent during both the decompression and compression processes. Since the training process of the first probabilistic prediction model 1Model remains consistent during compression and decompression, decompression can be achieved.

[0178] Then, for each second frequency level, the second compressed file Br for that second frequency level is decompressed using the same compression method as the second file Bc, for example, the second probabilistic prediction model 2Model and the second encoder 2coder, or the third encoder 3coder, to obtain the second file Bc corresponding to that second frequency level. It is worth noting that in practical applications, if the second probabilistic prediction model 2Model and the second encoder 2coder are used for compression, the second probabilistic prediction model 2Model needs to be initialized based on the initial model parameters of the second probabilistic prediction model 2Model in the second compressed file Br. This ensures that the initial model parameters of the second probabilistic prediction model 2Model are consistent during the decompression and compression processes. Since the training process of the second probabilistic prediction model 2Model remains consistent during compression and decompression, decompression can be achieved.

[0179] Finally, replace the corresponding replacement characters in the first file Ac with the characters in the second file Bc in sequence. The file obtained after all the characters in the second file Bc have been replaced is the target file.

[0180] Based on the data compression method provided above, the compression and decompression frameworks for the specific application of the data compression method are explained.

[0181] Figure 2 A schematic diagram illustrating a specific example of a data compression method provided for implementing the present invention. For example... Figure 2 As shown, the specific content includes:

[0182] First, the frequency of different characters in the target file to be compressed is statistically analyzed and sorted to obtain a sorted alphabet (corresponding to the third character table mentioned above). Then, based on the cutoff value, the sorted alphabet is divided into two categories: high-frequency characters and low-frequency characters. In the target file to be compressed, all low-frequency characters are replaced with token characters to obtain the first file Ac; in the target file to be compressed, all low-frequency characters are arranged in order to form the second file Bc. The first file Ac is probabilistically predicted using an independent RNN network model (as a specific example of the first probabilistic prediction model 1 mentioned above) and then encoded using entropy encoding (as a specific example of the encoding implemented by the first encoder 1 coder mentioned above) to obtain the first compressed file Ar. The second file Bc is probabilistically predicted using an independent small RNN network model (as a specific example of the second probabilistic prediction model 2 mentioned above) and then encoded using entropy encoding (as a specific example of the encoding implemented by the second encoder 2 coder mentioned above) to obtain the second compressed file Br. Since the probability of low-frequency characters appearing decreases, the second file Bc will be much smaller than the first file Ac; therefore, the second file Bc can use a small RNN network model. Furthermore, the first file Ac and the second file Bc can be processed using two GPUs (graphics processing units) to perform RNN probability prediction and encoding, accelerating the compression time. After obtaining the first compressed file Ar of the first file Ac and the second compressed file Br of the second file Bc, these are combined with the sorted alphabet (see the third character table above) to form the compressed file of the target file to be compressed. Finally, the cutoff, the first file Ac, the second file Bc, the first compressed file Ar, and other information are recorded in the metadata and added to the beginning of the compressed file of the target file to obtain the final compressed file.

[0183] Figure 3 yes Figure 2 A schematic diagram illustrating a specific example of the decompression method of the compression method shown.

[0184] like Figure 3As shown, based on the information recorded in the metadata, a sorted alphabet, a first compressed file "Ar", and a second compressed file "Br" are obtained. The sorted alphabet is then divided into high-frequency and low-frequency characters based on the cutoff, which are used to guide the decompression of the first compressed file "Ar" and the second compressed file "Br", respectively. After decompression to obtain the first file "Ac" and the second file "Bc", the characters in the second file "Bc" are used to replace the token characters in the first file "Ac". The resulting file is the target file, completing the decompression process. Similar to compression, the two GPU cards... Figure 2 The compression process uses RNN, small RNN, and entropy coding to decompress the first compressed file Ar and the second compressed file Br, respectively, effectively improving the decompression bandwidth.

[0185] Figure 4 This is a schematic diagram illustrating a specific example of a data compression method provided for implementing the present invention. Figure 2 The difference lies in the fact that after category fusion, the second file Bc is often very small. Therefore, instead of using the second probabilistic prediction model 2Model, entropy coding (as a specific example of the coding implemented by the third encoder 3coder mentioned above) can be directly used for compression. Since the coding is relatively simple, it can run on a CPU (central processing unit) card, effectively reducing the cost of using the algorithm.

[0186] Figure 5 for Figure 4 A schematic diagram illustrating a specific example of the decompression method shown in the compression diagram. (And...) Figure 3 The difference is that one GPU card and one CPU card are connected via... Figure 4 The compression process uses RNN and entropy coding to decompress the first compressed file Ar and the second compressed file Br, respectively, effectively improving the decompression bandwidth.

[0187] This compression and decompression method can effectively improve decompression bandwidth while ensuring almost no loss in compression ratio and reducing algorithm operating costs.

[0188] Figure 6 This is a schematic diagram illustrating a specific example of a data compression method provided for implementing the present invention. Figure 2 The difference lies in that, to improve the compression ratio, only the low-frequency character table is recorded. The high-frequency character table after decompression can be obtained by taking the complement of the full alphabet. Here, the low-frequency character table (corresponding to the fourth character table mentioned above) corresponds to the probability distribution of the small RNN output, and the order of characters in the low-frequency character table represents the character order represented by the probability distribution.

[0189] Figure 7 for Figure 6A schematic diagram illustrating a specific example of the decompression method shown in the compression diagram. (And...) Figure 3 The difference lies in that, based on the metadata, a low-frequency character table is obtained; based on the low-frequency character table, a complement is performed on the full alphabet to obtain a high-frequency character table, which is then decompressed. Here, the order of character numbers in the probability distribution output by the first probability prediction model 1 is consistent with the order of the full character arrangement under the character encoding type; for details, please refer to the description of specific example B above.

[0190] Figure 8 This is a schematic diagram illustrating a specific example of a data compression method provided for implementing the present invention. Figure 6 The difference lies in the fact that after category fusion, the second file Bc is often very small. Therefore, instead of using the second probabilistic prediction model 2Model, entropy coding (as a specific example of the coding implemented by the third encoder 3coder mentioned above) can be directly employed for compression. Since the coding is relatively simple, it can run on a CPU card, effectively reducing the cost of using the algorithm.

[0191] Figure 9 for Figure 8 A schematic diagram illustrating a specific example of the decompression method shown in the compression diagram. (And...) Figure 7 The difference is that the GPU card and CPU are connected via... Figure 8 The compression process uses RNN and entropy coding to decompress the first compressed file Ar and the second compressed file Br, respectively, effectively improving the decompression bandwidth.

[0192] This compression and decompression method can effectively improve decompression bandwidth while ensuring almost no loss in compression ratio and reducing algorithm operating costs.

[0193] Based on the same concept as the embodiments of the method of the present invention, the embodiments of the present invention also provide a data compression apparatus. The data compression apparatus includes several modules, each module being used to execute various steps in the data compression method provided in the embodiments of the present invention. The division of modules is not limited here. Those skilled in the art will clearly understand that in practical applications, the various steps in the data compression method provided in the embodiments of the present invention can be assigned to different modules as needed, that is, the internal structure of the apparatus can be divided into different modules to complete all or part of the functions described above. The modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the modules in the above apparatus can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0194] For example, the data compression apparatus is used to perform the data compression method provided in the embodiments of the present invention. Figure 10 This is a schematic diagram of the data compression device provided in an embodiment of this application. Figure 10 As shown, the data compression apparatus provided in this application embodiment includes:

[0195] The file acquisition module 1001 is used to acquire the target file to be compressed; wherein the target file includes m characters;

[0196] The character segmentation module 1002 is used to determine the occurrence frequency of each of the m characters in the target file; based on the occurrence frequency of each of the m characters, the m characters are classified into characters of at least two frequency levels; wherein, the at least two frequency levels are divided into a first frequency level and various second frequency levels below the first frequency level, and each second frequency level corresponds to a replacement character;

[0197] The file classification module 1003 is used to replace characters belonging to the second frequency level in the target file with corresponding replacement characters to obtain a first file corresponding to the first frequency level; and to assemble characters belonging to the second frequency level in the target file into a second file corresponding to the second frequency level in sequence.

[0198] The first compression module 1004 is used to perform a first compression on the first file and determine the first compressed file;

[0199] The second compression module 1005 is used to perform a second compression on the second file corresponding to each second frequency level, and determine the second compressed file.

[0200] The third compression module 1006 is used to determine the compressed file of the target file based on the first compressed file, each of the second compressed files, and the replacement characters of each second frequency level.

[0201] Exemplary electronic devices

[0202] Based on the same concept as the embodiments of the method of the present invention, the embodiments of the present invention also provide an electronic device. This electronic device is a data compression device. This electronic device can be a server or a terminal device. The terminal device can be a mobile phone, tablet computer, etc.

[0203] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0204] like Figure 11 As shown, the electronic device 1100 includes a processor 1101, a memory 1102, and a network interface 1103.

[0205] The processor 1101 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), GPUs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0206] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0207] For example, a computer program may be stored on the memory 1102, and when the processor 1101 executes the computer program, it implements the steps in the above-described data compression method embodiments, for example... Figure 1 The steps 101 to 108 are shown. Alternatively, when the processor 1101 executes the computer program, it implements the functions of each module in the above-described device embodiment. For example, the CPU executes steps 101 to 104, step 106, sends the first file to one GPU to execute step 105, sends the second file to another GPU to execute step 107, and then the CPU executes step 108; here, the two GPUs can compress the first and second files in parallel. Another example: the CPU executes steps 101 to 104, step 106, sends the first file to one GPU to execute step 105, and the CPU executes steps 107 and 108; here, the CPU and GPU can compress the first and second files in parallel.

[0208] For example, a computer program can be divided into one or more modules / units, which can be a series of computer program instruction segments capable of performing a specific function. One or more modules / units are stored in the memory 1102 and executed by the processor 1101 to complete the present invention. For instance, the computer program can be divided into a file acquisition module 1001, a character segmentation module 1002, a file classification module 1003, a first compression module 1004, a second compression module 1005, and a third compression module 1006. The specific functions of each module are described above.

[0209] The network interface 1103 is used to send and receive data, for example, to send data processed by the processor 1101 to other electronic devices, or to receive data sent by other electronic devices.

[0210] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 1100 relevant to this application are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1100 may include any other suitable components depending on the specific application. Furthermore, the electronic device may be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 1100 and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input devices, output devices, network access devices, buses, etc. For example, the input device may be a microphone array, and may also include, for example, a keyboard, mouse, etc. For example, the output device may output various information to the outside, and may include, for example, a display, speaker, printer, and communication network and its connected remote output devices, etc.

[0211] In addition to the methods, apparatus, and electronic devices described above, embodiments of this application may also provide a computer program product, comprising computer program instructions. When executed by a processor, these computer program instructions cause the processor to perform the steps of the data compression methods of the various embodiments of this application described in the "Methods" section of this specification. The computer program product may be written in any combination of one or more programming languages ​​to execute computer program code for performing the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The computer program code may be in source code form, object code form, executable file, or some intermediate form. The computer program code may be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0212] Furthermore, embodiments of this application may also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the data compression methods according to various embodiments of this disclosure as described in the "Method" section above. The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0213] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0214] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0215] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0216] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0217] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0218] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0219] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0220] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0221] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

[0222] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A data compression method, characterized in that, Applied to electronic devices, including: Obtain the target file to be compressed; wherein the target file includes m characters; Determine the frequency of occurrence of each of the m characters in the target file; Based on the occurrence frequency of each of the m characters, the m characters are classified into characters of at least two frequency levels; wherein, the at least two frequency levels are divided into a first frequency level and various second frequency levels below the first frequency level, and each second frequency level corresponds to a replacement character; Replace the characters in the target file that belong to the second frequency level with the corresponding replacement characters to obtain the first file corresponding to the first frequency level; Perform a first compression on the first file to determine the first compressed file; The characters belonging to the second frequency level in the target file are arranged in order to form a second file corresponding to the second frequency level; For each second file corresponding to a second frequency level, the second file is subjected to a second compression to determine the second compressed file; The compressed file of the target file is determined based on the first compressed file, each of the second compressed files, and the replacement characters of each second frequency level.

2. The method according to claim 1, characterized in that, The first compression of the first file includes: The first file is compressed based on a first probability prediction model and a first encoder; wherein, the first probability prediction model takes one character or multiple consecutive characters in the first file as the first model input and outputs a first probability distribution of p characters in the first file, and the first encoder encodes the first probability distribution to obtain the encoding result of the next character input by the first model in the first file.

3. The method according to claim 2, characterized in that, The electronic device includes a first graphics processor, which runs the first probability prediction model and the first encoder to compress the first file.

4. The method according to claim 2, characterized in that, The model parameters of the first probability prediction model change during the first compression of the first file; or, The first probability prediction model is trained based on the first file.

5. The method according to any one of claims 2 to 4, characterized in that, The second compression of the second file includes: The second file is compressed based on a second probability prediction model and a second encoder. The number of model parameters of the second probability prediction model is less than that of the first probability prediction model. The second probability prediction model takes one character or multiple consecutive characters in the second file as input and outputs a second probability distribution of q characters in the second file. The second encoder encodes the second probability distribution to obtain the encoding result of the next character input by the second model in the second file.

6. The method according to claim 5, characterized in that, The electronic device includes a second graphics processor that runs the second probability prediction model and the second encoder to compress the second file.

7. The method according to claim 5, characterized in that, The model parameters of the second probabilistic prediction model change during the second compression process of the second file; or, The second probabilistic prediction model is trained based on the second file.

8. The method according to claim 1, characterized in that, The second compression of the second file includes: The second file is subjected to a second compression based on the frequency of occurrence of different characters in the third encoder and the second file.

9. The method according to claim 8, characterized in that, The electronic device includes a central processing unit that runs the third encoder to perform a second compression on the second file.

10. A data compression device, characterized in that, include: An acquisition module is used to acquire the target file to be compressed; wherein the target file includes m characters; The statistics module is used to determine the frequency of occurrence of each of the m characters in the target file; The classification module is used to classify the m characters into characters of at least two frequency levels based on the occurrence frequency of each of the m characters; wherein the at least two frequency levels are divided into a first frequency level and various second frequency levels below the first frequency level, and each second frequency level corresponds to a replacement character; The file determination module is used to replace characters belonging to the second frequency level in the target file with corresponding replacement characters to obtain a first file corresponding to the first frequency level; and to assemble characters belonging to the second frequency level in the target file into a second file corresponding to the second frequency level in sequence. The first compression module is used to perform a first compression on the first file and determine the first compressed file; The second compression module is used to perform a second compression on the second file corresponding to each second frequency level, and determine the second compressed file. An integration module is used to determine the compressed file of the target file based on the first compressed file and each of the second compressed files.

11. A data compression device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-9.

12. A data compression device, characterized in that, The device executes computer program instructions to perform the method as described in any one of claims 1-9.

13. A computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

14. A computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

Images