Data compression method and device, equipment, storage medium and program product

Abstract

The invention relates to a data compression method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a reference data set and a first initial data stream; the reference data set comprises a corresponding relationship between a plurality of reference data and replacement codes; the first data length of the replacement code is smaller than the second data length of the reference data corresponding to the replacement code; determining replaceable target data from the first initial data stream according to each reference data in the reference data set; and according to each replacement code in the reference data set, performing replacement processing on the target data in the first initial data stream to obtain a compressed first target data stream. By adopting the method, the storage space occupied by the data can be reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data compression method, device, equipment, storage medium and program product. Background Art

[0002] Data storage is an important process in data processing scenarios. For example, with the growth of global data traffic and the expansion of cloud technology usage, more and more users choose to transfer data to cloud services through the network for storage and processing. For the storage side, it is crucial to effectively reduce the storage space occupied by data. Summary of the invention

[0003] Based on this, it is necessary to provide a data compression method, device, equipment, storage medium and program product that can reduce the storage space occupied by data in order to address the above technical problems.

[0004] In a first aspect, the present application provides a data compression method. The method comprises:

[0005] Acquire a reference data set and a first initial data stream; the reference data set includes a correspondence between a plurality of reference data and a replacement code; a first data length of the replacement code is smaller than a second data length of the reference data corresponding to the replacement code;

[0006] Determining replaceable target data from the first initial data stream according to each reference data in the reference data set;

[0007] According to each replacement code in the reference data set, a replacement process is performed on the target data in the first initial data stream to obtain a compressed first target data stream.

[0008] In one of the embodiments, the process of constructing the reference data set includes: receiving at least one reference data submitted by a user, and generating a replacement code for each reference data, obtaining a correspondence between at least one reference data and the replacement code, and adding each correspondence to the reference data set; and / or obtaining at least one candidate data stream; determining at least one reference data from the candidate data stream; generating a corresponding replacement code for the reference data, obtaining a correspondence between the reference data and the replacement code, and adding the correspondence between the reference data and the replacement code to the reference data set.

[0009] In one embodiment, determining at least one reference data from the candidate data stream includes: determining a target character string that meets a preset characteristic condition from each character string according to data characteristics of each character string in the candidate data stream; and using the target character string as the reference data.

[0010] In one of the embodiments, replacement processing is performed on the target data in the first initial data stream according to each replacement code in the reference data set, including: determining the target position of the target data in the first initial data stream; replacing the target data at the target position in the first initial data stream using the replacement code corresponding to the reference data identical to the target data.

[0011] In one of the embodiments, the method further includes: acquiring a second initial data stream obtained by compression processing; the second initial data stream includes at least one target replacement code; and performing replacement processing on the target replacement code in the second initial data stream according to the reference data set to obtain a decompressed second target data stream.

[0012] In one of the embodiments, the method further includes: performing secondary compression processing on the first target data stream based on a lossless compression algorithm; and storing the first target data stream obtained by the secondary compression processing.

[0013] In a second aspect, the present application also provides a data compression device. The device comprises:

[0014] An acquisition module, used to acquire a reference data set and a first initial data stream; the reference data set includes a correspondence between a plurality of reference data and a replacement code; a first data length of the replacement code is less than a second data length of the reference data corresponding to the replacement code;

[0015] A determination module, configured to determine replaceable target data from the first initial data stream according to each of the reference data in the reference data set;

[0016] A replacement module is used to perform replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set to obtain a compressed first target data stream.

[0017] In one of the embodiments, the acquisition module is specifically used to: receive at least one reference data submitted by a user, and generate a replacement code for each reference data, obtain a correspondence between at least one reference data and the replacement code, and add each correspondence to the reference data set; and / or obtain at least one candidate data stream; determine at least one reference data from the candidate data stream; generate a corresponding replacement code for the reference data, obtain a correspondence between the reference data and the replacement code, and add the correspondence between the reference data and the replacement code to the reference data set.

[0018] In one embodiment, the acquisition module is specifically used to: determine a target character string that meets a preset characteristic condition from each character string according to data characteristics of each character string in the candidate data stream; and use the target character string as the reference data.

[0019] In one embodiment, the replacement module is specifically used to: determine the target position of the target data in the first initial data stream; and replace the target data at the target position of the first initial data stream using a replacement code corresponding to reference data identical to the target data.

[0020] In one of the embodiments, the device also includes a decompression module, which is used to: obtain a second initial data stream obtained by compression processing; the second initial data stream includes at least one target replacement code; according to the reference data set, perform replacement processing on the target replacement code in the second initial data stream to obtain a decompressed second target data stream.

[0021] In one of the embodiments, the device further includes a secondary compression module, which is used to: perform secondary compression processing on the first target data stream based on a lossless compression algorithm; and store the first target data stream obtained by the secondary compression processing.

[0022] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect above.

[0024] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0025] The above-mentioned data compression method, device, equipment, storage medium and program product obtain a reference data set and a first initial data stream including a correspondence between a plurality of reference data and replacement codes; determine replaceable target data from the first initial data stream according to each reference data in the reference data set; perform replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set to obtain a compressed first target data stream. Since the first data length of the replacement code is less than the second data length of the reference data corresponding to the replacement code, the replacement processing is performed on the target data based on the replacement code, and the data length of the obtained first target data stream is less than the data length of the first initial data stream. Therefore, the storage space occupied by storing the first target data stream is greatly reduced, which is equivalent to reducing the storage space occupied by the first initial data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a flow chart of a data compression method in one embodiment;

[0028] Figure 2 A schematic diagram of a process for constructing a reference data set in one embodiment;

[0029] Figure 3 A schematic diagram of a process for determining a target character string in one embodiment;

[0030] Figure 4 A schematic diagram of a process flow of a knowledge base generation phase in one embodiment;

[0031] Figure 5 A schematic diagram of a process of performing a replacement process on target data in one embodiment;

[0032] Figure 6 is a flow chart of a data compression stage in one embodiment;

[0033] Figure 7 A schematic diagram of a two-stage data compression process in one embodiment;

[0034] Figure 8 A schematic diagram of a process of decompressing a data stream in one embodiment;

[0035] Fig. 9 A schematic diagram of a process of two-stage data decompression in one embodiment;

[0036] Fig.10 A schematic diagram of the connection relationship of units for executing the data compression method in an embodiment;

[0037] Fig.11 A schematic diagram showing a comparison of data lengths before and after compression using various compression algorithms in one embodiment;

[0038] Fig.12 is a structural block diagram of a data compression device in one embodiment;

[0039] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It should be understood that many specific details are set forth in the following description to facilitate a full understanding of the present application, but the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0042] It will be understood that “at least one” means one or more, and “plurality” means two or more.

[0043] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0044] Data compression is the process of encoding data to reduce its storage space. Through data compression, more efficient data storage, transmission and processing can be achieved. With the growth of global data traffic and the expansion of the commercial scale of cloud technology, more and more users choose to transmit data to the cloud through the network for storage and processing. Therefore, the use of data compression technology before data transmission and storage is conducive to reducing the latency of users uploading or downloading data, reducing the storage overhead required for data, and improving the utilization efficiency of resources. To ensure the validity of data, cloud device providers can only perform lossless compression on data. Current compression algorithms are limited to syntactic compression, that is, only compressing the character representation of data. However, these methods lack understanding of the content of the data, and therefore lose some compression efficiency.

[0045] In view of this, the embodiment of the present application provides a data compression method with compression / decompression isomorphism assisted by a knowledge base, which utilizes the front-to-back correlation of data character representation in a specific field to establish a knowledge base shared by the transmitting and receiving ends, reduce the character representation length of the relevant data, and further combine the grammatical compression algorithm to achieve higher compression efficiency, so as to achieve efficient storage and transmission of data and reduce the time delay of user upload or download. At the same time, the compression method proposed in the embodiment of the present application is isomorphic in the compression and decompression process, and the same algorithm can be used to complete compression and decompression, reducing the deployment cost of the compression algorithm.

[0046] It should be noted that the data compression method provided in the embodiment of the present application may be implemented as a data compression device, which may be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware. Optionally, the computer device may be a device in a cloud platform. The computer device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and servers, etc. The Internet of Things devices may be smart TVs, smart car-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers. The following is an example of the application of this method to a computer device.

[0047] In one embodiment, Figure 1 As shown, a data compression method is provided, comprising the following steps:

[0048] Step 101: Acquire a reference data set and a first initial data stream.

[0049] The first initial data stream is a data stream to be compressed, for example, a data stream to be transmitted to other devices or stored after compression.

[0050] Optionally, as mentioned above, the computer device may be a device in a cloud platform, and therefore, the first initial data stream may be a data stream submitted to the cloud platform by a user using the cloud platform. For ease of understanding, the following embodiments are described below using the scenario of processing a data stream in a cloud platform as an example.

[0051] The reference data set includes a correspondence between a plurality of reference data and replacement codes.

[0052] The first data length of the replacement code is smaller than the second data length of the reference data corresponding to the replacement code.

[0053] Optionally, a reference data set is pre-stored in the computer device.

[0054] Optionally, the computer device acquires the reference data set synchronously when acquiring the first initial data stream. For example, the computer device receives the first initial data stream sent by the sending end and the reference data set sent by the sending end.

[0055] In the embodiment of the present application, the character type of the replacement code may be different from the character type of the reference data. In this way, after the reference data is replaced with the replacement code to perform compression processing, it is easy to identify the replacement code from the compressed data stream during decompression, thereby ensuring the success rate of decompression.

[0056] Step 102: determining replaceable target data from the first initial data stream according to each reference data in the reference data set.

[0057] The reference data may be a string. The first initial data stream may include multiple strings. Therefore, based on each reference data in the reference data set, it is possible to determine which strings in the first initial data stream are identical to the reference data, and then use the determined strings as target data, so that replacement processing can be performed on the target data.

[0058] Optionally, the determined target data may be single or multiple.

[0059] It is understandable that when there are multiple target data, some of the target data may be repeated. In other words, some target data may appear multiple times in the first initial data stream.

[0060] It is understandable that, in some cases, there may be no replaceable target data in the first initial data stream.

[0061] Step 103: Perform replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set to obtain a compressed first target data stream.

[0062] For each target data, the target data is replaced in the first initial data stream using a replacement code corresponding to the same reference data as the target data. In this way, all target data can be replaced based on each replacement code.

[0063] It can be understood that, in addition to the target data, the first initial data stream also contains non-target data. Therefore, the compressed first target data stream includes replacement codes and non-target data.

[0064] It can be understood that in some cases, there is only at least one target data in the first initial data stream, and no non-target data exists. In this case, all target data can be replaced to achieve maximum compression.

[0065] The above data compression method obtains a reference data set including a correspondence between a plurality of reference data and replacement codes and a first initial data stream; determines replaceable target data from the first initial data stream according to each reference data in the reference data set; and performs replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set to obtain a compressed first target data stream. Since the first data length of the replacement code is less than the second data length of the reference data corresponding to the replacement code, the data length of the first target data stream obtained by performing replacement processing on the target data based on the replacement code is less than the data length of the first initial data stream. Therefore, the storage space occupied by storing the first target data stream is greatly reduced, which is equivalent to reducing the storage space occupied by the first initial data stream.

[0066] In one embodiment, the process of constructing a reference data set includes: receiving at least one reference data submitted by a user, generating a replacement code for each reference data, obtaining a correspondence between at least one reference data and the replacement code, and adding each correspondence to the reference data set.

[0067] The reference data set may be deployed in the computer device. Optionally, the reference data set is empty during initial construction. The user may submit multiple reference data at any time or when submitting the first initial data stream.

[0068] The computer device generates a replacement code for each reference data to obtain multiple sets of corresponding relationships, and adds each corresponding relationship to the reference data set to obtain an updated reference data set.

[0069] In one embodiment, the computer device can also determine the correspondence between at least one set of reference data and replacement codes based on the data stream stored in the storage space and / or the first initial data stream, and add the corresponding relationship between the reference data and replacement codes to the reference data set. Figure 2 A schematic diagram of a process of constructing a reference data set is shown. The process of constructing a reference data set includes:

[0070] Step 201: Obtain at least one candidate data stream.

[0071] The at least one candidate data stream may be at least one data stream pre-stored in the storage space of the cloud platform. Alternatively, the currently acquired first initial data stream may be used as a candidate data stream.

[0072] Step 202: determine at least one reference data from the candidate data streams.

[0073] Taking the case where there are multiple candidate data streams as an example, for each candidate data stream, it is determined whether reference data can be extracted from the candidate data stream. In this way, at least one reference data is obtained by extracting each candidate data stream.

[0074] The reference data are not repeated.

[0075] In one embodiment, determining at least one reference data from the candidate data stream includes: determining a target string that meets a preset characteristic condition from each string according to data characteristics of each string in the candidate data stream; and using the target string as reference data.

[0076] That is, for each candidate data stream, the computer device can determine the data features of each string in the candidate data stream. For each string, according to the data features of the string, it is determined whether the string meets the preset feature conditions. If it meets, the string can be used as reference data, otherwise, the string is not used as reference data. In this way, multiple reference data can be extracted.

[0077] Exemplarily, the content of data stream A is: "data compression is important for cloudstorage. Data compression reduces the storage overhead of data on cloudstorage servers.", then "data", "compression", "important", etc. are all strings in data stream A.

[0078] Optionally, the data features may be determined based on multiple character string attributes of each character string. In this way, it is determined based on the multi-dimensional data features whether it meets the preset feature conditions, thereby ensuring the validity of the determined reference data.

[0079] In one embodiment, the string attributes of a string include the frequency of occurrence of the string in the candidate data stream and the data length of the string.

[0080] For example, Figure 3 A schematic diagram of a process for determining a target string is shown. According to the data features of each string in the candidate data stream, a target string that meets the preset feature conditions is determined from each string, including:

[0081] Step 301, determine the product of the data length and frequency of each character string to obtain the data feature of each character string.

[0082] For example, the length of "data" is 4 and its frequency in data stream A is 3, then the data feature of "data" is 12.

[0083] Step 302: Arrange the data features of each character string according to a preset sorting method to obtain a sequence value of each character string.

[0084] The preset sorting method can be descending or ascending, which is not specifically limited here.

[0085] Step 303: determine a character string whose sequence value is greater than or equal to a preset sequence value as a target character string.

[0086] In other words, the top N character strings with the largest data features may be determined as target character strings.

[0087] For example, the data feature of string 1 is 12, the data feature of string 2 is 22, the data feature of string 3 is 2, the data feature of string 4 is 9, the data feature of string 5 is 3, and the data feature of string 6 is 10. Taking descending order as an example, the order is: string 2, string 1, string 6, string 4, string 5, string 3. Correspondingly, the order value of string 2 is 1, the order value of string 1 is 2, and the others are similar.

[0088] A preset sequence value is preset in the computer device. For example, if the preset sequence value is 3, the target character strings are character string 2, character string 1, and character string 6.

[0089] In this way, since the data characteristics of each character string are obtained according to the data length and frequency of each character string, and the top N character strings with the largest data characteristics are taken as the target character strings, it means that the character strings with higher frequency and longer length are determined as the target character strings. In this way, when the target character string appears in the first initial data stream, it is replaced, which can shorten the data length of the first initial data stream to the greatest extent and achieve maximum compression processing.

[0090] Step 203: Generate a corresponding replacement code for the reference data, obtain a corresponding relationship between the reference data and the replacement code, and add the corresponding relationship between the reference data and the replacement code to the reference data set.

[0091] The replacement codes corresponding to the reference data are different.

[0092] The computer device can automatically generate a corresponding replacement code for each reference data.

[0093] Optionally, the replacement code may be a special symbol, such as "+", "-", "*", " / ", etc., to ensure that it does not appear in the data, or appears very rarely.

[0094] Optionally, the replacement code is a character type that will no longer appear in the first data stream.

[0095] If the generated reference dataset can be stored in the storage space of the cloud platform, it can be called upon during compression.

[0096] Optionally, the reference dataset can be deployed synchronously on the sending end and the receiving end.

[0097] Optionally, the newly added candidate data stream in the cloud platform can be read periodically or upon receiving an update instruction from the upper layer, and new reference data can be generated based on the new candidate data stream. A replacement code can be generated for the new reference data to obtain multiple new corresponding relationships, and the new corresponding relationships can be added to the reference data set to update the reference data set.

[0098] In the embodiment of the present application, by establishing a reference data set, the target data is replaced with a short-length replacement code to reduce its character representation length, thereby compressing the storage space occupied by the data stream.

[0099] For ease of understanding, based on the above content, the process of generating a reference data set is fully described in an exemplary manner. It should be noted that the knowledge base here is the reference data set, the knowledge data is the reference data, and the code is the replacement code. Figure 4 A schematic diagram of a process of the knowledge base generation stage is shown.

[0100] Step 1: Determine whether the user-defined knowledge data list is received; if so, execute step 2; if not, execute step 3.

[0101] Step 2: Read user input. Read the knowledge data file provided by the user, which includes multiple knowledge data. Output the number of knowledge data N, and execute step 6.

[0102] Step 3: Statistical parameter setting: Set the maximum character length L of knowledge data and the number N of knowledge data.

[0103] Step 4: Word frequency statistics. Read the data stream pre-stored by the user, and count the number of occurrences T(i) of the string with a length L(i)<=L in all data streams, where i represents the index of the string.

[0104] Step 5: Knowledge data screening. Select N knowledge data according to the rules. The rules may be as follows: select N strings with the largest product of string length and number of occurrences L(i)×T(i) as knowledge data.

[0105] Step 6: Code generation. Generate N different codes of length L. C A short-length code using special symbols.

[0106] Step 7: Write the knowledge base. Write the knowledge data and code into the storage space in the form shown in Table 1, where each row of knowledge data and code corresponds one to one.

[0107] Table 1

[0108] Knowledge data (i.e. reference data) Code (i.e. replacement code) K(1) C(1) K(2) C(2) ... ... K(N) C(N)

[0109] After the reference data set is generated and stored, the user enters the data compression phase when uploading or downloading data. For example, when a user uploads a data stream to the cloud platform for storage, the computer device obtains the data stream uploaded by the user and compresses the data stream; when a user downloads a data stream from the cloud platform, the computer device reads the data stream, decompresses it, and feeds it back to the user.

[0110] The compression and decompression process is described below.

[0111] In one embodiment, Figure 5 A schematic diagram of a process of performing a replacement process on target data is shown. According to each replacement code in a reference data set, a replacement process is performed on the target data in a first initial data stream, including:

[0112] Step 501: determine the target position of target data in the first initial data stream.

[0113] When there are multiple target data, each target position corresponding to each target data is determined.

[0114] It can be understood that when a certain target data appears multiple times in the first initial data stream, multiple target positions corresponding to the target data can be determined.

[0115] For example, the target positions of "data" in data stream A are "1-4", "50-53", and "99-102".

[0116] Step 502: Replace the target data at the target position of the first initial data stream using the replacement code corresponding to the reference data identical to the target data.

[0117] That is, at the target position, the target data and the replacement code are interchanged to implement the replacement process.

[0118] Optionally, in order to reduce the complexity of the algorithm, the computer device synchronously determines the position of each reference data and each replacement code in the first initial data stream during the compression process. At this time, the position of each reference data (i.e., target data) in the first initial data stream can be obtained, while the position of the replacement code is empty. During the decompression process, for the data stream to be decompressed, the position of each reference data and each replacement code in the data stream to be decompressed can also be synchronously determined. At this time, the position of the target replacement code in the data stream to be decompressed can be obtained, while the position of the reference data is empty. In this way, whether compression or decompression, the same processing can be executed to determine the position and perform the interchange process, thereby reducing the complexity of the algorithm, that is, providing a data compression method with compression / decompression isomorphism.

[0119] In one embodiment, the method further includes: performing secondary compression processing on the first target data stream based on a lossless compression algorithm; and storing the first target data stream obtained by the secondary compression processing.

[0120] In other words, the above compression processing is a one-time compression process. To ensure the compression quality, in the embodiment of the present application, a secondary compression processing is also provided, that is, a secondary compression processing is performed on the first target data stream using a lossless compression algorithm to obtain the final first target data stream.

[0121] Optionally, the lossless compression algorithm is a grammatical compression algorithm.

[0122] Optionally, the lossless compression algorithm is a dictionary compression coding algorithm. Dictionary compression coding includes LZ77 (Lempel-Ziv, 1977) algorithm, LZ78 (Lempel-Ziv, 1978) algorithm and LZW (Lempel-Ziv-Welch) algorithm.

[0123] Optionally, the lossless compression algorithm is the LZW algorithm.

[0124] In this way, the data length of the data stream after the secondary compression processing is shorter than the data length after the primary compression processing, thereby further reducing the storage space and improving the compression efficiency.

[0125] For ease of understanding, based on the above description and Figure 4 The compression process is described as an example. Figure 6 A schematic diagram of a process flow of a data compression stage is shown.

[0126] Step 1: Knowledge base reading: Read out the knowledge data K(i) and its corresponding code C(i) shown in Table 1.

[0127] Step 2: Data reading: Read the data stream to be compressed.

[0128] Step 3: Search. Search the location of each knowledge data K(i) and code C(i) in the data stream and record their locations respectively, where the location of the knowledge data is recorded as LOC_k(i) and the location of the code is recorded as LOC_c(i).

[0129] Step 4: Bidirectional replacement. In the data stream to be compressed, the knowledge data and the code are replaced with each other according to the searched position, that is, K(i) at LOC_k(i) is replaced with C(i), and C(i) at LOC_c(i) is replaced with K(i).

[0130] Since the data compression stage uses bidirectional replacement, it is only necessary to perform bidirectional replacement again in the decompression stage. In addition, the data compression stage only processes the character segment of the knowledge data. In order to further combine syntax compression to reduce data redundancy and storage space, a two-level data compression process is provided. Figure 7 A schematic flow chart of a two-stage data compression process is shown.

[0131] Step 1: Data compression phase. Execute as follows Figure 6 The data compression phase flow is shown.

[0132] Step 2: Syntax compression: Use a syntax-level lossless compression algorithm to compress the replaced data.

[0133] In one embodiment, Figure 8 A schematic diagram of a process of decompressing a data stream is shown. The method also includes:

[0134] Step 801: Acquire a second initial data stream obtained through compression processing.

[0135] The second initial data stream includes at least one target replacement code.

[0136] Optionally, the second initial data stream is a data stream required by the user. Exemplarily, after receiving the data acquisition request submitted by the user, the computer device acquires the second initial data stream from the storage space.

[0137] It can be understood that the second initial data stream is a compressed data stream, and therefore includes at least one target replacement code.

[0138] Step 802: Perform a replacement process on the target replacement code in the second initial data stream according to the reference data set to obtain a decompressed second target data stream.

[0139] Among them, the target replacement code existing in the second initial data stream can be determined according to each replacement code in the reference data set. In this way, the target replacement code can be replaced in the second initial data stream using the reference data corresponding to the target replacement code to obtain the decompressed second target data stream.

[0140] In one embodiment, a replacement process is performed on a target replacement code in a second initial data stream according to a reference data set to obtain a decompressed second target data stream, including: determining a candidate position of each target replacement code in the second initial data stream; and replacing the target replacement code at the candidate position in the second initial data stream using reference data corresponding to the target replacement code in the reference data set.

[0141] When there are multiple target replacement codes, candidate positions corresponding to the target replacement codes are determined.

[0142] It can be understood that when a certain target replacement code appears multiple times in the second initial data stream, multiple candidate positions corresponding to the target replacement code can be determined.

[0143] In this way, at the candidate position, the target replacement code and the corresponding reference data are interchanged to implement the replacement process.

[0144] In one embodiment, the method further includes: performing a decompression process on the second initial data stream based on a lossless compression algorithm; performing a replacement process on the target replacement code in the second initial data stream after the decompression process according to a reference data set to obtain a decompressed second target data stream.

[0145] Among them, if the second initial data stream is the data obtained after a compression process is performed once and then a second compression process is performed based on a calculation-free compression algorithm, then when decompressing, it is necessary to first decompress the second initial data stream based on the lossless compression algorithm, and then perform replacement processing on the target replacement code to achieve secondary decompression processing.

[0146] Here, the decompression algorithm is an algorithm for decompressing data of the lossless compression algorithm.

[0147] In the embodiment of the present application, since there is an isomorphic characteristic between compression and decompression, only one algorithm needs to be deployed in the computer device to implement the compression and decompression process, which has a lower deployment cost. In addition, since the reference data set is only shared by the sending and receiving ends, a third party cannot correctly decode the transmission information, which improves the security of data transmission. On this basis, by combining the grammatical compression algorithm, a higher compression efficiency is achieved to achieve efficient storage and transmission of data and reduce the latency of user upload or download.

[0148] For example, Fig. 9 A schematic diagram of a two-stage data decompression process is shown.

[0149] Step 1: Decompress the syntax. Use the syntax compression decompression algorithm to restore the data before syntax compression.

[0150] Step 2: Data compression phase. Execute as follows Figure 6 The data compression phase process shown in FIG. 4 is used to implement decompression.

[0151] For ease of understanding, the knowledge base-assisted isomorphic data compression method provided in the embodiment of the present application is described below with a complete embodiment.

[0152] The implementation of this method includes three stages: knowledge base generation stage, data compression stage and data decompression stage. In the knowledge base generation stage, a knowledge base containing information such as knowledge data and codes is generated and shared with the devices at both ends of the transmission and reception; in the data compression stage, the knowledge base is used to perform database-assisted data compression; in the data decompression stage, the knowledge base is used to execute the compression algorithm again to achieve data decompression.

[0153] The knowledge base generation stage includes the following steps: custom judgment of knowledge data, reading of user input, setting of statistical parameters, word frequency statistics, knowledge data screening, code generation, and knowledge base writing.

[0154] The data compression stage includes the following steps: knowledge base reading, data reading, searching, and bidirectional replacement. Optionally, the data compression stage can also combine the existing lossless compression technology to perform grammatical compression, and in this case, an additional step is added after the existing steps: grammatical data compression.

[0155] In the data decompression stage, the following steps are included: knowledge base reading, data reading, searching, and bidirectional replacement. Optionally, when combined with syntax compression, an additional step is added before the existing stage: syntax data decompression.

[0156] like Fig.10 The schematic diagram of the connection relationship of the units for executing the data compression method is shown. The data compression method provided in the embodiment of the present application is executed by three units: a knowledge base generation unit, a knowledge base reading and writing unit, and a data compression unit. Each unit can be deployed in a computer device.

[0157] Among them, the knowledge base generation unit performs operations in the knowledge base generation stage, generates structured data and transmits it to the knowledge base read-write units at both ends of the sending and receiving ends; the knowledge base read-write unit performs write operations in the knowledge base generation stage, writes structured data into the storage space, performs read operations in the data compression stage and the data decompression stage, and provides the structured data reading to the data compression unit for calling; the data compression unit performs operations in the data compression stage or the data decompression stage, reads the knowledge data and code information in the knowledge base and replaces them.

[0158] Knowledge base generation phase: used to generate a knowledge base shared by the sender and receiver, which contains information such as knowledge data and code. Figure 4 shown.

[0159] After the knowledge base is generated and stored, users enter the data compression phase when uploading or downloading data.

[0160] Exemplarily, in one embodiment, due to the correlation between data contents, a knowledge base can be established to assist data compression. Take the data to be transmitted as "data compression is important for cloud storage. Data compression reduces the storage overhead of data on cloud storage servers." as an example. In one implementation, the knowledge data is customized by the user, for example, the user defines the knowledge data as "data", "compression", "cloud" and "storage". In the knowledge base generation stage, the following steps are performed:

[0161] First determine whether the knowledge data is user-defined. If the judgment result is yes, read the user input, where the knowledge data provided by the user is "data", "compression", "cloud", "storage", and the number of knowledge data N = 4. Generate a code for each knowledge data, that is, generate 4 short-length codes with a length of 1: "+", "-", "*", " / ". Write the generated code and knowledge data into the knowledge base accordingly. The corresponding relationship is shown in Table 2:

[0162] Table 2

[0163] Knowledge Data Code data + compression - cloud * storage /

[0164] During the data compression phase, the following steps are performed:

[0165] Read the knowledge base and the data stream to be compressed. For example, read the data stream A: "data compression is important for cloud storage. Data compression reduces the storage overhead of data on cloud storage servers." Determine the location of the knowledge data and code in the knowledge base in data stream A, as shown in Table 3. Null indicates that there is no corresponding data in the data stream.

[0166] Table 3

[0167]

[0168] Then replace the knowledge data at the given position of LOC k with the code, and replace the code at the given position of LOC c with the knowledge data, and get the compression result of data stream A: "+-is imported for * / .+-reduces the / overhead of+on* / servers."

[0169] After compression, the length of the data is reduced from 128 characters to 73 characters, greatly reducing the storage space.

[0170] In the data decompression stage, due to the compression / decompression isomorphism characteristics of the compression method provided by the present application, the same steps as those in the data compression stage are performed, as follows:

[0171] The knowledge base is read, and the data stream to be decompressed is read, for example, the data stream to be decompressed is "+-isimportant for* / .+-reduces the / overhead of+on* / servers.". According to the knowledge base, the positions of each knowledge data and code in the data stream are determined, as shown in Table 4. Null indicates that there is no corresponding data in the data stream.

[0172] Table 4

[0173]

[0174] Then a two-way replacement process is performed, that is, the knowledge data at the given position of LOC_k is replaced with code, and the code at the given position of LOC_c is replaced with knowledge data to obtain the decompressed data: "Data compression is important for cloud storage. Data compression reduces the storage overhe ad of data on cloud storage servers.".

[0175] In one embodiment, when the user does not specify knowledge data, the computer device can automatically generate a knowledge base. For example, the data currently stored by the user in the cloud is "data compression is important for cloud storage. Data compression reduces the storage overhead of data on cloudstorage servers." The following steps are performed in the knowledge base generation stage:

[0176] Determine whether the knowledge data is customized. If the result is no, set the maximum character length of each knowledge data L = 20 and the number of knowledge data N = 4. Then read the data stored by the user: "data compression is important for cloud storage. Data compression reduces the storage overhead of data on cloud storage servers." The number of occurrences of each string is shown in Table 5.

[0177] Table 5

[0178] Knowledge Data Data length L(i) Frequency T(i) L(i)×T(i) Sorted order data 4 3 12 3 compression 11 2 22 1 is 2 1 2 important 9 1 9 for 3 1 3 cloud 5 2 10 4 storage 7 3 21 2 reduces 7 1 7 the 3 1 3 overhead 8 1 8 of 2 1 2 on 2 1 2 servers 7 1 7

[0179] Arrange the strings in descending order according to the product of their length and number of occurrences, select the first four strings "compression", "storage", "data", and "cloud" as knowledge data, and generate codes for each string, that is, generate four short-length codes of length 1, "-", " / ", "+", and "*". Write to the knowledge base.

[0180] In one embodiment, the data compression method provided in the embodiment of the present application can be further combined with a syntax compression algorithm to perform two-level data compression to further reduce storage space. Still taking the data stream "data compression is important for cloud storage. data compression reduces the storage overhead of data on cloud storage servers." as an example, the generation result of its knowledge base refers to the above description.

[0181] In the data compression stage, the data compression process is first executed based on the knowledge base to compress / decompress the homogeneously compressed data: "+-is important for* / .+-reduces the / overhead of+on* / server s.", with a length of 73 characters.

[0182] Then the dictionary compression encoding LZW algorithm in the lossless compression algorithm is used to compress the syntax of "+-is important for* / .+-reduces the / overhead of+on* / servers." The result is "+- * / . ", with a length of 58 characters, further reduces storage space and improves compression efficiency.

[0183] In the data decompression stage, the LZW decompression algorithm is used to * / . ” Decompress and get the data “+-is important for* / .+-reduces the / overhead of+on* / servers.”. Continue to execute the data compression process and restore the data “datacompression is important for cloud storage.data co mpression reduces thestorage overhead of data on cloud storage servers.”.

[0184] like Fig.11 The figure shows the data length comparison before and after compression by various compression algorithms. age. ", which is 91 symbols in length. The knowledge base-assisted compression / decompression isomorphic compression algorithm provided in the embodiment of the present application has better performance than LZW compression, and the length of the compressed characters is 73. Further applying LZW compression on the basis of the knowledge base-assisted compression / decompression isomorphic compression algorithm provided in the present application can further compress the data to 58 characters.

[0185] As can be seen from the above, the compression algorithm based on the knowledge base can replace a large amount of repeated, long-string knowledge data with short-string codes from the perspective of data content based on the knowledge base shared by the sender and receiver, thereby reducing the storage space required for the data and improving the security of data transmission. The algorithm is implemented in three stages: the first stage is the knowledge base generation stage, the second stage is the data compression stage, and the third stage is the data decompression stage. For compression algorithms with compression / decompression isomorphism, the compression and decompression process flows are consistent and the algorithms are the same. Therefore, for the sender and receiver devices, only one data compression unit needs to be deployed for the compression algorithm, which reduces the deployment cost of the compression algorithm. The algorithm ensures the consistency of the compression and decompression process by bidirectionally replacing long-transmitted knowledge data with short-string codes. Since the knowledge data and the code are one-to-one corresponding, the data can be restored after an even number of replacements, thereby achieving compression / decompression isomorphism. In the existing compression algorithms, there is no feature of compression / decompression isomorphism. Because it is a two-way replacement data compression, different from the traditional compression method, in the compression process of the present invention, the data and code in the knowledge base are replaced in two directions, that is, the knowledge data in the original data is replaced by the corresponding code, and the code in the original data is replaced by the knowledge data. Since the knowledge data is repeated in large quantities and has a long length, and the code appears less frequently and is shorter, the total length of the data is reduced after the two-way replacement, thereby compressing the storage space of the data. In the existing compression scheme, the replacement of data and the corresponding code is unidirectional, that is, only the characters in the data can be converted into the corresponding code, which means that all data need to be converted. The two-way replacement only focuses on the given string and the corresponding code.

[0186] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0187] Based on the same inventive concept, the embodiment of the present application also provides a data compression device for implementing the data compression method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data compression device embodiments provided below can refer to the limitations on the data compression method above, and will not be repeated here.

[0188] In one embodiment, Fig.12 As shown, a data compression device is provided, and the data compression device 1200 includes: an acquisition module 1201, a determination module 1202 and a replacement module 1203, wherein:

[0189] The acquisition module 1201 is used to acquire a reference data set and a first initial data stream; the reference data set includes a correspondence between a plurality of reference data and a replacement code; the first data length of the replacement code is less than the second data length of the reference data corresponding to the replacement code;

[0190] A determination module 1202, configured to determine replaceable target data from the first initial data stream according to each of the reference data in the reference data set;

[0191] The replacement module 1203 is configured to perform replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set to obtain a compressed first target data stream.

[0192] In one embodiment, the acquisition module 1201 is specifically used to: receive at least one reference data submitted by a user, and generate a replacement code for each reference data, obtain a correspondence between at least one reference data and the replacement code, and add each correspondence to the reference data set; and / or obtain at least one candidate data stream; determine at least one reference data from the candidate data stream; generate a corresponding replacement code for the reference data, obtain a correspondence between the reference data and the replacement code, and add the correspondence between the reference data and the replacement code to the reference data set.

[0193] In one embodiment, the acquisition module 1201 is specifically used to: determine a target string that meets a preset characteristic condition from each string according to the data characteristics of each string in the candidate data stream; and use the target string as the reference data.

[0194] In one embodiment, the replacement module 1203 is specifically used to: determine the target position of the target data in the first initial data stream; and replace the target data at the target position of the first initial data stream using a replacement code corresponding to reference data identical to the target data.

[0195] In one of the embodiments, the device also includes a decompression module, which is used to: obtain a second initial data stream obtained by compression processing; the second initial data stream includes at least one target replacement code; according to the reference data set, perform replacement processing on the target replacement code in the second initial data stream to obtain a decompressed second target data stream.

[0196] In one of the embodiments, the device further includes a secondary compression module, which is used to: perform secondary compression processing on the first target data stream based on a lossless compression algorithm; and store the first target data stream obtained by the secondary compression processing.

[0197] Each module in the above data compression device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0198] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data compression data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data compression method is implemented.

[0199] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0201] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0202] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0203] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0204] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data compression method, characterized in that: The method comprises: Acquire a reference data set and a first initial data stream; the reference data set includes a correspondence between a plurality of reference data and replacement codes; a first data length of the replacement code is smaller than a second data length of the reference data corresponding to the replacement code; Determining replaceable target data from the first initial data stream according to each of the reference data in the reference data set; According to each replacement code in the reference data set, a replacement process is performed on the target data in the first initial data stream to obtain a compressed first target data stream.

2. The method according to claim 1, characterized in that The process of constructing the reference data set includes: receiving at least one reference data submitted by a user, generating a replacement code for each reference data, obtaining a correspondence between at least one reference data and the replacement code, and adding each correspondence to the reference data set; and / or, Obtain at least one candidate data stream; determining at least one reference data from the candidate data streams; Generate a corresponding replacement code for the reference data, obtain a corresponding relationship between the reference data and the replacement code, and add the corresponding relationship between the reference data and the replacement code to the reference data set.

3. The method according to claim 2, characterized in that The determining at least one reference data from the candidate data stream comprises: Determining a target character string that meets a preset characteristic condition from each character string according to data characteristics of each character string in the candidate data stream; The target character string is used as the reference data.

4. The method according to any one of claims 1 to 3, characterized in that: The performing replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set comprises: Determining a target position of the target data in the first initial data stream; The target data is replaced at the target position of the first initial data stream using a replacement code corresponding to reference data identical to the target data.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a second initial data stream obtained by compression processing; the second initial data stream includes at least one target replacement code; According to the reference data set, a replacement process is performed on the target replacement code in the second initial data stream to obtain a decompressed second target data stream.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Performing secondary compression processing on the first target data stream based on a lossless compression algorithm; The first target data stream obtained by the secondary compression process is stored.

7. A data compression device, characterized in that: The device comprises: An acquisition module, configured to acquire a reference data set and a first initial data stream; the reference data set includes a correspondence between a plurality of reference data and a replacement code; a first data length of the replacement code is smaller than a second data length of the reference data corresponding to the replacement code; A determination module, configured to determine replaceable target data from the first initial data stream according to each of the reference data in the reference data set; A replacement module is used to perform replacement processing on the target data in the first initial data stream according to each replacement code in the reference data set to obtain a compressed first target data stream.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images