A data compression method, device, equipment and computer storage medium

Abstract

The application discloses a data compression method, which comprises the following steps: selecting part of data in reference data, and splitting the part of data to obtain a first character string set; obtaining at least one to-be-compressed character string from to-be-compressed data, and comparing the fingerprint of the at least one to-be-compressed character string in a fingerprint set corresponding to the first character string set; in the case that no matching item is obtained, expanding the part of data by a preset step length, splitting the expanded part of data to obtain a second character string set, comparing the fingerprint of the at least one to-be-compressed character string in a fingerprint set corresponding to the second character string set, and recording the offset and length of the matching item in the reference data into difference data in the case that the matching item is obtained. When the above method is used for compressing to-be-compressed data, the compression efficiency can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technology field, and in particular to a data compression method, device, equipment and computer storage medium. BACKGROUND

[0002] Delta compression is a technology of realizing data compression by eliminating redundant data. Delta compression is suitable for compression of similar data. For example, data block A is similar to data block B. By means of delta calculation, the delta data Δ of data block A relative to data block B can be obtained. B,A Since the repeated data between data block A and data block B is not contained in the delta data Δ B,A , the data amount of the delta data Δ B,A is much smaller than that of data block A. Therefore, the cost of storing or transmitting the delta data Δ B,A is much smaller than that of storing or transmitting data block A.

[0003] However, the person skilled in the art finds that there is still a problem of low compression efficiency when using the delta compression technology to compress data blocks. SUMMARY

[0004] The embodiments of the present application disclose a data compression method, device, equipment and computer storage medium, which can effectively improve the efficiency of data compression.

[0005] In a first aspect, the present application provides a data compression method, which comprises the following steps:

[0006] selecting part of data in the reference data, and splitting the part of data to obtain a first character string set;

[0007] obtaining at least one to-be-compressed character string from the to-be-compressed data, and comparing the at least one to-be-compressed character string in the first character string set;

[0008] in a case where no matching character string is obtained, expanding the part of data in the reference data by a preset step length, splitting the expanded part of data to obtain a second character string set, comparing the at least one to-be-compressed character string in the second character string set, and in a case where a matching character string is obtained, recording the offset and length of the matching character string in the reference data into the delta data.

[0009] According to the above scheme, the device obtains a first string set by splitting part of the reference data, and then compares at least one to-be-compressed string obtained by splitting the to-be-compressed data with the first string set by using the fingerprints of the strings. When none of the at least one to-be-compressed string matches any string in the first string set, the device expands the part of the reference data to obtain a second string set, and then compares the at least one to-be-compressed string with the second string set by using the fingerprints of the strings. When any string in the at least one to-be-compressed string matches any string in the first string set, the device records the offset and length of the matched string in the reference data in the difference data. This process is repeated until the compression of the to-be-compressed data is completed. It can be seen that, since the device selects to split part of the reference data in the above method, when no matched string is obtained, the part of the reference data is expanded and split again instead of being directly split in its entirety, so that the above method reduces the operations of calculating the fingerprints of the strings and determining whether the strings match by comparing the fingerprints of the strings, and thus greatly reduces the time cost of compressing the to-be-compressed data and improves the speed of compressing the to-be-compressed data.

[0010] In some possible designs, the above method further includes: comparing the at least one to-be-compressed string in the second string set, and recording, in the difference data, a non-matched string and a length of the non-matched string when no matched string is obtained. Specifically, when no matched string is obtained, the device can continue to split the to-be-compressed data to obtain at least one new to-be-compressed string, and compress the newly split to-be-compressed string to obtain a matched string. Therefore, before the device records the offset and length of the matched string in the reference data in the difference data, the device records the non-matched string and the length of the non-matched string in the difference data.

[0011] It can be seen that, when the amount of data of the non-matched string is too large, the above method can reduce the number of times of writing the non-matched string and the length of the non-matched string into the difference data, and also reduces the number of times of recovering the non-matched string from the difference data during decompression, thereby improving the efficiency of data compression and decompression.

[0012] In some possible designs, after the at least one to-be-compressed string is compared in the first string set, the above method further includes: recording, in the difference data, an offset and a length of a matched string in the reference data when the matched string is obtained.

[0013] In some possible designs, after recording the offset and length of the matching string in the reference data into the difference data, the method further includes: in a case where the last byte in the matching string is the last byte of the to-be-compressed data, calculating an amount of data of the non-matching string and a length of the non-matching string in the difference data; and in a case where the amount of data is greater than a first threshold, re-compressing the to-be-compressed data.

[0014] It can be seen that, when the last byte in the matching string is the last byte of the to-be-compressed data, it indicates that the device has completed compression of the to-be-compressed data. However, it can be known from the foregoing that the device only splits part of the data in the reference data, and therefore there can be a case where the non-matching to-be-compressed string matches the part of the data in the reference data that is not split, and therefore there is part of the data in the difference data that can be compressed. In order to avoid this case, the present application determines whether there is data in the difference data that can be continuously compressed by judging an amount of data occupied by the non-matching string and its length in the difference data after obtaining the difference data, and if there is, the to-be-compressed data is re-compressed, thereby improving the saving of the space occupied by the difference data in the memory of the device and the overhead required when transmitting the difference data.

[0015] In some possible designs, before the part of the data is enlarged by the preset step size, the method further includes: determining whether a split condition is met, and in a case where the split condition is met, enlarging the part of the data by the preset step size, where the split condition includes that a number of at least one to-be-matched string is less than or equal to a first preset number, and a number of strings in the first string set is less than or equal to a second preset number. It can be seen that, only when the split condition is met, the device enlarges the part of the data by the preset step size, thereby improving the efficiency of data compression.

[0016] In some possible design, after the at least one to-be-compressed string is compared in the second string set, the method further includes: obtaining a first offset and a second offset, where if a matched string is obtained after the at least one to-be-compressed string is compared in the second string set, the first offset is an offset of a last byte in the matched string in the to-be-compressed data, and the second offset is an offset of a last byte in the matched string in the reference data or an offset of a last byte in the partial data in the reference data, specifically, the second offset is a maximum value of the offset of the last byte in the matched string in the reference data and the offset of the last byte in the expanded partial data in the reference data; if no matched string is obtained, the first offset is an offset of a last byte in the at least one to-be-compressed string in the to-be-compressed data, and the second offset is an offset of a last byte in the partial data in the reference data. In a case where a ratio of the first offset and the second offset is less than a second threshold, a target to-be-compressed string is segmented from the to-be-compressed data; the target to-be-compressed string is compared in the second string set, and in a case where a matched string is obtained, an offset of the matched string in the reference data and a length of the matched string are recorded in the difference data.

[0017] In some possible design, the method further includes: in a case where the ratio of the first offset and the second offset is greater than or equal to the second threshold, a target to-be-compressed string is segmented from the to-be-compressed data, the partial data is expanded by a fixed step, and the expanded partial data is segmented to obtain a third string set; the target to-be-compressed string is compared in the third string set, and in a case where a matched string is obtained, an offset of the matched string in the reference data and a length of the matched string are recorded in the difference data.

[0018] It can be seen that, the present application determines how to continue to segment the to-be-compressed data and the reference data by judging whether the ratio of the first offset and the second offset is less than the second threshold, so as to ensure that a proper amount of strings are segmented from the reference data, so as to improve a probability of successful matching of the to-be-compressed string and the string segmented from the reference data. It can be understood that, when the probability of successful matching of the to-be-compressed string and the string segmented from the reference data is high, a data amount of an unmatched string and a length of the unmatched string in the difference data is small, and then a probability of re-compression of the to-be-compressed data by the device is reduced, so as to improve the efficiency of data compression.

[0019] In some possible designs, the matching string includes the to-be-compressed string and / or an enlarged to-be-compressed string, the enlarged to-be-compressed string being obtained by enlarging the matching to-be-compressed string, so that the enlarged to-be-compressed string is included in both the reference data and the to-be-compressed data. As can be seen, when the to-be-compressed string matches the string segmented from the reference data, the matching string is obtained by enlarging the to-be-compressed string, thereby reducing the matching operation of the device on the subsequent bytes of the matching to-be-compressed string, and improving the data compression efficiency.

[0020] In a second aspect, the present application provides a data compression device, which includes a segmentation unit, a matching unit, and a recording unit,

[0021] The segmentation unit is configured to select part of data in the reference data, and segment the part of data to obtain a first string set.

[0022] The segmentation unit is further configured to obtain at least one to-be-compressed string from the to-be-compressed data.

[0023] The matching unit is configured to compare the at least one to-be-compressed string in the first string set.

[0024] The segmentation unit is further configured to, in a case where the matching string is not obtained, enlarge the part of data in the reference data by a preset step, and segment the enlarged part of data to obtain a second string set.

[0025] The matching unit is configured to compare the at least one to-be-compressed string in the second string set.

[0026] The recording unit is configured to, in a case where the matching string is obtained, record an offset and a length of the matching string in the reference data in the difference data.

[0027] In some possible designs, after the matching unit compares the at least one to-be-compressed string in the second string set, the recording unit is further configured to, in a case where the matching string is not obtained, record a non-matching string and a length of the non-matching string in the difference data.

[0028] In some possible designs, after the matching unit compares the at least one to-be-compressed string in the first string set, the recording unit is further configured to, in a case where the matching string is obtained, record an offset and a length of the matching string in the reference data in the difference data.

[0029] In some possible design, the apparatus further includes a re-compression unit, configured to: in a case where the last byte in the matched string is the last byte of the to-be-compressed data, calculate the data amount of the unmatched string in the differential data and the length of the unmatched string; and in a case where the data amount is greater than the first threshold, re-compress the to-be-compressed data.

[0030] In some possible design, the splitting unit is further configured to determine whether a splitting condition is met, and in a case where the splitting condition is met, expand the partial data by a preset step length, wherein the splitting condition includes that the number of the at least one to-be-matched string is less than or equal to a first preset number, and the number of the strings in the first string set is less than or equal to a second preset number.

[0031] In some possible design, the splitting unit is further configured to obtain a first offset and a second offset, wherein the first offset is an offset of the last byte in the matched string in the to-be-compressed data, or an offset of the last byte in the at least one to-be-compressed string in the to-be-compressed data; and the second offset is an offset of the last byte in the matched string in the reference data, or an offset of the last byte in the partial data in the reference data; and the splitting unit is further configured to split the target to-be-compressed string from the to-be-compressed data in a case where a ratio of the first offset to the second offset is less than a second threshold; the matching unit is further configured to compare the target to-be-compressed string in the second string set; and the recording unit is further configured to record, in a case where the matched string is obtained, the offset and the length of the matched string in the reference data into the differential data.

[0032] In some possible design, the splitting unit is further configured to split the target to-be-compressed string from the to-be-compressed data in a case where a ratio of the first offset to the second offset is greater than or equal to the second threshold, and expand the partial data by a fixed step length, and split the expanded partial data to obtain a third string set; the matching unit is further configured to compare the target to-be-compressed string in the third string set; and the recording unit is further configured to record, in a case where the matched string is obtained, the offset and the length of the matched string in the reference data into the differential data.

[0033] In some possible design, the matched string includes the to-be-compressed string and / or the expanded to-be-compressed string, and the expanded to-be-compressed string is obtained by expanding the matched to-be-compressed string, so that the reference data and the to-be-compressed data both include the expanded to-be-compressed string.

[0034] In a third aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores code, and the processor executes the code in the memory to implement all or part of the steps of the first aspect.

[0035] In a fourth aspect, the present application provides a computer readable storage medium, which comprises instructions, when the instructions are executed on an electronic device, causing the electronic device to perform all or part of the steps of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced.

[0037] Figure 1 is a schematic diagram of data compression provided by the present application;

[0038] Figure 2A is a schematic diagram of obtaining a matching string provided by the present application;

[0039] Figure 2B is a schematic diagram of obtaining a matching string provided by the present application;

[0040] Figure 3 is a flowchart of a subsequent processing method provided by the present application in the case of obtaining a matching string;

[0041] Figure 4 is a flowchart of a subsequent processing method provided by the present application in the case of not obtaining a matching string;

[0042] Figure 5 is a schematic diagram of a heuristic segmentation of a data block B provided by the present application;

[0043] Figure 6 is a schematic diagram of a specific embodiment provided by the present application;

[0044] Figure 7 is a structural schematic diagram of a data compression device provided by the present application;

[0045] Figure 8 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0047] In order to facilitate understanding of the technical solutions of the present application, the application scenarios involved in the present application will be briefly described first.

[0048] With the advent of the big data era, the explosive growth of data volume has brought enormous challenges to data storage and transmission. Data compression is crucial for improving the utilization of device storage space and the transmission efficiency of network bandwidth. Among these technologies, differential compression is widely used because it can effectively eliminate redundant data.

[0049] by Figure 1 For example, Figure 1 A schematic diagram is shown of compressing data to be compressed using reference data, wherein the data to be compressed is similar to the reference data.

[0050] In this application, "similarity between the data to be compressed and the reference data" means that a portion of the data to be compressed is the same as a portion of the data in the reference data, while another portion of the data to be compressed is different from another portion of the data in the reference data. In this application, the device extracts features from the data to be compressed and the reference data, then compares the features of the data to be compressed with those of the reference data to determine that the data to be compressed and the reference data have some similar features and some different features; that is, the data to be compressed and the reference data contain identical data. At this point, the reference data can be used to compress the data to be compressed. The specific process is as follows:

[0051] Step 11: Split all data in the baseline data to obtain x strings (i.e., strings B1, B2, ..., B...). x ).

[0052] A string is a finite sequence of at least one character. Strings include numbers, letters, punctuation marks, etc., and therefore a string contains one or more bytes.

[0053] In one specific embodiment, the device can segment the baseline data according to a preset length, or it can segment the baseline data based on content, etc., without specific limitations. It should be understood that when the device segments the baseline data according to the preset length, at least x-1 strings (i.e., strings B1, B2, ..., B...) will exist among the x segments. x-1 The length of string B is the preset length. x The length of each string is less than or equal to the preset length. When the device segments the baseline data based on the content, the lengths of the resulting x strings are not entirely the same.

[0054] For example, assuming the reference data is "cloud, rain, snow and frost", when the device splits the reference data according to the preset length (the preset length is 3), the reference data can be split into 3 strings, which are "cloud, rain", ", snow and", and "frost". When the device splits the reference data B based on the content, the data block B can be split into 5 strings, which are "cloud", ",", "rain", ",", "snow and frost".

[0055] Step 12: starting from the first byte in the data to be compressed, a string, i.e., string A1, is split from the data to be compressed according to the splitting manner of the reference data.

[0056] In a specific implementation, if the device splits the reference data based on the preset length, the device also splits the string A1 from the data to be compressed based on the preset length, where the length of the string A1 is the preset length. If the device splits the reference data based on the content, the device also splits the string A1 from the data to be compressed based on the content.

[0057] Step 13: the string A1 is matched with the strings B1, B2,..., Bn. x If the string A1 matches any one of the strings B1, B2,..., Bn, steps 14-15 are performed; if the string A1 does not match any one of the strings B1, B2,..., Bn, step 16 is performed. x x

[0058] In a specific implementation, the device calculates the fingerprints of the strings B1, B2,..., Bnand the string A1 by using a hash function, respectively, to obtain fingerprints F1, F2,..., Fnand a fingerprint f1. The fingerprints F1, F2,..., Fnare used to identify the strings B1, B2,..., Bn, respectively, and the fingerprint f1 is used to identify the string A1. The device saves the fingerprints F1, F2,..., Fninto a fingerprint set. Then, the device matches the fingerprint f1 with the fingerprints in the fingerprint index. In the case that the fingerprint f1 matches any one of the fingerprints in the fingerprint index (assuming the fingerprint is Fk), the bytes in the string A1 are matched with the bytes in the string Bkidentified by the fingerprint Fkone by one, so as to determine that the data in the string A1 is completely identical with the data in the string Bk. x x x x x s s s s s s ​​​​​​​​​​​​Match; if the data in string A1 matches the data in string B... s If the data in strings A1 and B are not completely identical, it means that strings A1 and B1 are not identical. i Mismatch. If fingerprint f1 does not match any of the fingerprints in the fingerprint index, it means that string A1 does not match strings B1, B2, ..., B. x None of the strings in A1 match. Understandably, when string A1 and string B... i During matching, it indicates that the baseline data also includes string A1; when string A1 matches strings B1, B2, ..., B... x If none of the strings in the reference data match, it means that string A1 is not included in the reference data.

[0059] Step 14: Obtain the matching string A1′ corresponding to string A1, and record the offset of matching string A1′ in the baseline data and the length of matching string A1′ in the difference data.

[0060] In one example, the device can obtain the matching string A1′ as follows: In the data to be compressed, the device obtains the first byte after string A1 (denoted as byte a1), and in the base data, the device obtains string B. s The first byte after the first byte (denoted as byte b1) is then compared with byte b1. If byte a1 and byte b1 are the same, the second byte after string A1 (denoted as byte a2) is retrieved from the data to be compressed, and the device retrieves string B from the base data. s The second byte (denoted as byte b2) is then compared with byte a2. This process continues until a different byte is found, or the currently compared byte is the last byte in the data to be compressed, or the currently compared byte is the last byte in the reference data, resulting in the matching string A1'. The matching string A1' includes the string A1 and the aforementioned identical bytes. Alternatively, if byte a1 is different from byte b1, then the matching string A1' is the string A1.

[0061] For example, such as Figure 2A As shown, in the data to be compressed, the first byte after string A1 is byte a1, the second byte is byte a2, and the third byte is byte a3; in the reference data, string B... s The first byte after that is byte b1, the second byte is byte b2, and the third byte is byte b3. Assume that byte a1 is the same as byte b1, byte a2 is the same as byte b2, and byte a3 is different from byte b3. Then, in determining the relationship between string A1 and string B... iAfter matching, the device first compares byte a1 with byte b1 and determines that byte a1 and byte b1 are the same. Then, it compares byte a2 with byte b2 and determines that byte a2 and byte b2 are the same. Finally, it compares byte a3 with byte b3 and determines that byte a3 and byte b3 are different. At this point, the matched string A1′ includes string A1, byte a1, and byte a2.

[0062] For example, such as Figure 2B As shown, in the data to be compressed, the first byte after string A1 is byte a1, the second byte is byte a2, and byte a2 is the last byte in the data to be compressed; in the reference data, string B... s The first byte after that is byte b1, the second byte is byte b2, and the third byte is byte b3. Assume byte a1 is the same as byte b1, and byte a2 is the same as byte b2. Then, in determining the relationship between string A1 and string B... i After matching, the device first compares byte a1 with byte b1 and determines that byte a1 and byte b1 are the same. Then, it compares byte a2 with byte b2 and determines that byte a2 and byte b2 are the same. Finally, since byte a2 is the last byte in the data to be compressed, the comparison will stop and the matching string A1′ will be obtained. The matching string A1′ includes string A1, byte a1 and byte a2.

[0063] In one specific embodiment, after obtaining the matching string A1′, the device generates a copy record based on the matching string A1′ and writes the copy record into the differential data. This copy record includes the offset of the matching string A1′ in the base data and the length of the matching string A1′.

[0064] Step 15: Starting from the first byte after the matched string A1′, continue to split the data block to be compressed according to the splitting method of the base data to obtain the string A2.

[0065] Step 16: Record the length of string A1 and the difference data of string A1.

[0066] In one specific embodiment, the device generates an insertion record based on the string A1 and writes the insertion record into the differential data. The insertion record includes the length of string A1 and the string A1 itself.

[0067] Step 17: Starting from the first byte after string A1, continue to split the data block to be compressed according to the splitting method of the base data to obtain string A3.

[0068] Step 18: Referring to the above steps 13-17, the subsequent strings (for example, string A2, string A3) segmented from the data block A are processed until the compression of the data to be compressed is completed, so as to obtain the differential data.

[0069] It can be seen that, before the technical solution shown in the prior art compresses the data to be compressed, all the data in the reference data need to be segmented to obtain x strings, and then the fingerprints of the x strings are calculated respectively, so as to establish the complete fingerprint set of the reference data. Long-term practice proves that the time consumed for establishing the complete fingerprint set of the reference data is the main time consumption in the whole process of compressing the data to be compressed, so the efficiency of using the technical solution shown in the prior art to compress the data to be compressed is relatively low. Figure 1 The technical solution shown in the prior art has the following disadvantages: Figure 1 The technical solution shown in the prior art has the following disadvantages:

[0070] In order to further improve the compression efficiency of the data to be compressed, the present application provides a data compression method. Using the method to compress the data to be compressed can greatly shorten the compression time, thereby improving the compression efficiency. The data compression method provided by the present application will be described in detail below. Figures 3-5 The data compression method provided by the present application will be described in detail below. First, please refer to Figure 3 , Figure 3 The technical solution shown in the prior art has the following disadvantages:

[0071] S101: Selecting part of the data in the reference data and segmenting the part of the data to obtain a first string set.

[0072] In a specific implementation, the device selects part of the data in the reference data, and then segments the part of the data into m strings using a content-based string segmentation method (for example, Rabin Hash algorithm, Gear Hash algorithm, etc.), so as to obtain the first string set. The first string set includes strings B1, B2, …, B m , m≥1 and m is a positive integer.

[0073] S102: Obtaining at least one string to be compressed from the data to be compressed, and comparing the at least one string to be compressed in the first string set. If a matching string is obtained, S103 is executed; if no matching string is obtained, S104 is executed.

[0074] In a specific implementation, the device segments the data to be compressed in the same way as the reference data is segmented, to obtain n strings to be compressed, i.e., strings A1, A2, …, A n , n≥1 and n is a positive integer. Then, the strings A1, A2, …, A nA comparison is performed in the first string set to determine if a matching string exists. The matching string includes either the string to be compressed or an amplified version of the string to be compressed. The amplified version is obtained by amplifying the matching string to be compressed, ensuring that both the baseline data and the data to be compressed include the amplified version.

[0075] In one specific embodiment, strings A1, A2, ..., A are respectively... n The first string set is compared to determine if a matching string exists. The specific process is as follows: Fingerprint algorithms (e.g., cyclic redundancy check, xxHash algorithm, etc.) are used to calculate strings A1, A2, ..., A... n The fingerprints f1, f2, ..., f1 are obtained by combining the fingerprints of the first string set with the fingerprints of each string in the first string set. n And the fingerprint set corresponding to the first string set. Where fingerprint f1 is the fingerprint of string A1, fingerprint f2 is the fingerprint of string A2, ..., fingerprint f... n For string A n The fingerprint set corresponding to the first string set includes fingerprints F1, F2, ..., F1. m Fingerprint F1 is the fingerprint of string B1, fingerprint F2 is the fingerprint of string B2, ..., fingerprint F m For string B m The fingerprints. Then, the fingerprints f1, f2, ..., f... n The fingerprints corresponding to the first string set are compared to determine if a matching string exists.

[0076] In a more specific embodiment, using fingerprint f i For example, 1≤i≤n and i is a positive integer, the fingerprint f i The fingerprints F1, F2, ..., F1 in the fingerprint set corresponding to the first string set are respectively... m Perform a comparison; if the fingerprint f i With any fingerprint in the fingerprint set (let's say fingerprint F) t If the fingerprint f matches (1≤t≤m and t is a positive integer), then the fingerprint f will be... i The corresponding string A i Data and fingerprints in F t The corresponding string B t The data in string A is compared byte by byte. i Data and string B in t Given that the data in the two strings are exactly the same, determine string A. i With string B t Match. If fingerprint fi If it does not match any fingerprint in the fingerprint set, then string A is determined. i It does not match any of the strings in the first string set. Understandably, fingerprint f i A match with any fingerprint in the fingerprint set means that there exists a fingerprint in the fingerprint set that matches fingerprint f. i The same fingerprint. Fingerprint f i "Not matching any fingerprint in the fingerprint set" means that there is no fingerprint in the fingerprint set that matches fingerprint f. i The same fingerprints.

[0077] In a more specific embodiment, in fingerprint f i With any fingerprint in the fingerprint set (let's say fingerprint F) t If a match is found, the fingerprint f will be... i The corresponding string A i Data and fingerprints in F t The corresponding string B t The data in string A is compared byte by byte. The specific process is as follows: String A is compared byte by byte. i The first byte in the string B t Compare the first byte of string A, if string A i The first byte in the string B t If the first byte of each string is the same, then string A will be... i The second byte in the string B t The second byte in string A is compared; if string A... i The second byte in the string B t The second byte is the same as the second byte in string A. i The third byte in the string B t The third byte in the string is compared, and so on, until string A is compared. i The last byte in the string B t The last byte in the string is compared to determine the string A. i The last byte in the string B t Are the last bytes of strings A and B the same? i All bytes and string B t When all bytes in the string correspond to the same value, determine string A. i Data and string B in t The data in them are exactly the same, thus determining string A. i With string B t Match. Or, if the currently compared bytes are two different bytes, stop the comparison and determine string A. i With string Bt Not match.

[0078] In a specific embodiment, after determining that the string A i matches the string B t , the device performs an amplification process on the string A i to obtain a matching string corresponding to the string A i . The specific process is as follows: in the data to be compressed, the device obtains the first byte (denoted as byte al) after the string A i , and in the reference data, the device obtains the first byte (denoted as byte bl) after the string B t . Then, the byte al is compared with the byte bl. If the byte al is the same as the byte bl, in the data to be compressed, the second byte (denoted as byte a2) after the string A i is obtained, and in the reference data, the second byte (denoted as byte b2) after the string B t is obtained. Then, the byte b2 is compared with the byte b2. In this way, the comparison is continued until a different byte is found, or the current compared byte is the last byte in the data to be compressed, or the current compared byte is the last byte in the reference data, thereby obtaining the matching string, which includes the string A i and the same bytes as described above. In addition, if the byte al is different from the byte bl, the matching string is the string A i . It can be understood that the specific process of the device performing the amplification process on the string A i to obtain the matching string corresponding to the string A i is similar to the specific process of obtaining the matching string al' in step 14 in the foregoing content, and can be referred to in the foregoing content. Figure 2A and Figure 2B .

[0079] It is worth noting that the application adopts a content-based string segmentation method to segment the reference data and the data to be compressed, which can ensure the same string segmentation point at the same content position, making it possible to match the strings involved in the subsequent step S104. It should also be noted that since the same fingerprint may be calculated for different strings using the fingerprint algorithm, after determining that the fingerprints of two strings match, it is necessary to compare whether the data in the two strings is the same to determine whether the two strings match. In addition, the application determines whether two strings match based on the fingerprints of the strings, because the fingerprints of the strings can identify the strings. In the application, it is first determined whether the fingerprints of two strings are the same, and if the fingerprints of the two strings are the same, the data of the two strings is compared. If the data of the two strings is also the same, the two strings match. Compared with directly comparing the data of two strings to determine whether the two strings match, the method in the application can shorten the time to determine whether two strings match, thereby improving the data compression efficiency.

[0080] S103: In the case where no matching string is obtained, the partial data is enlarged by a preset step size, and the enlarged partial data is segmented to obtain a second string set. At least one string to be compressed is compared in the second string set.

[0081] In the embodiment of the application, in the case where no matching string is obtained, the device determines that the strings A1, A2,..., A n do not match the strings B1, B2,..., B m In this case, the device will enlarge the partial data of the reference data by a preset step size, and segment the enlarged partial data to obtain m+l strings, thereby obtaining a second string set, wherein the second string set includes the strings B1,..., B m ,..., B m+l , l≥m and l is a positive integer. Then, the strings A1, A2,..., A n are compared in the second string set.

[0082] (1) If a matching string is obtained, the offset of the matching string in the reference data and the length of the matching string are recorded in the difference data. It can be understood that after the device records the offset of the matching string in the reference data and the length of the matching string in the difference data, it will continue to compress the data in the data to be compressed that has not been compressed. The above content (i.e., how the device performs the compression operation in the case where a matching string is obtained) will be described in detail in steps S201-S209 below.

[0083] (2) If no matching string is obtained, the non-matching string and the length of the non-matching string are recorded into the difference data. It can be understood that after the device records the non-matching string and the length of the non-matching string into the difference data, the device will continue to compress the data in the to-be-compressed data that has not been compressed. The above content (i.e., how the device performs the compression operation in the case of obtaining a non-matching string) will be described in detail in steps S301-S307 below.

[0084] In a specific embodiment, the preset step length is the total length of one string, and there is a linear relationship between l and m (for example, l = 2 x m). Specifically, the device can expand the partial data to obtain the second string set in the following manner: in step S101, the device has divided the partial data into m strings, and therefore the device only needs to continue to divide the reference data from the first byte after the partial data until l strings (B m+1 ,B m+2 ,…,B m+l ) are divided, at which time the m+l strings constitute the second string set. It should be understood that there can be other non-linear relationships between l and m (for example, l = m 2 , l = 2 m , etc.), which are not limited specifically herein.

[0085] In a specific embodiment, before expanding the partial data by the preset step length, the device first determines whether a division condition is met, and when the division condition is met, the device expands the partial data of the reference data by the preset step length. The division condition includes that the number of the at least one to-be-compressed string is less than or equal to a first preset number, and the number of strings in the first string set is less than or equal to a second preset number. It can be understood that the number of the at least one to-be-compressed string in the division condition being less than or equal to the first preset number means that the number of the at least one to-be-compressed string continuously divided from the to-be-compressed data is less than or equal to the first preset number, and any one of the at least one to-be-compressed string continuously divided from the to-be-compressed data does not match any one of the strings in the first string set.

[0086] The first preset number and the second preset number can be set by a user or dynamically adjusted by the device according to a data amount of the string currently segmented from the data to be compressed and a data amount of the partial data of the reference data, which is not specifically limited here. The condition of “the number of strings in the first string set being less than or equal to the second preset number” means that the number of strings segmented from the partial data after continuous expansion of the partial data for multiple times is less than or equal to the second preset number. Specifically, the device can obtain m+l strings after first expanding the partial data, m+l+k strings after second expanding the partial data, and m+l+k+u strings after third expanding the partial data, and the number of strings segmented from the partial data after the continuous expansion for three times is m+l+k+u, where k and u are positive integers. Therefore, in actual application, the device can also determine whether the segmentation condition is met by judging whether the partial data is continuously expanded and the number of times of continuously expanding the partial data. For example, it is assumed that the number of at least one string to be compressed is less than the first preset number, the device first expands the partial data, and m+l strings are segmented from the partial data after the first expansion. If any one of the at least one string to be compressed does not match any one of the m+l strings, the device second expands the partial data, and m+l+k strings are segmented from the partial data after the second expansion. If any one of the at least one string to be compressed does not match any one of the m+l+k strings, the device third expands the partial data, and m+l+k+u strings are segmented from the partial data after the third expansion. Similarly, if any one of the at least one string to be compressed does not match the string segmented from the partial data after the rth expansion, the device determines that the segmentation condition is not met, and thus the partial data is not continuously expanded. It can be understood that u≥k≥l≥m, and u, k, and l have a linear relationship with m, for example, u=4×m, k=3×m, and l=2×m. u, k, and l can also have a nonlinear relationship with m, or u and k have a nonlinear relationship with m, and l has a linear relationship with m, which is not specifically limited here.

[0087] It is worth noting that if no matched string is obtained in the comparison of the at least one character to be compressed in the first string set, and the segmentation condition is not met, the device will continue to segment the data to be compressed, so as to obtain a new string to be compressed (i.e., string A n+1 ). Then, the string A n+1 is compared in the first string set, and if the string An+1 If the string A matches any string in the first string set, the string A is first obtained n+1 corresponding matching string, and then subsequent operations are performed according to the processes described in S201-S209. If the string A n+1 If the string A does not match any string in the first string set, subsequent operations are performed according to the processes described in S301-S307.

[0088] S104: In the case where the matching string is obtained, the offset of the matching string in the reference data and the length of the matching string are recorded in the difference data.

[0089] In a specific embodiment, after the at least one to-be-compressed string is compared in the first string set, in the case where the matching string is obtained, the device performs subsequent operations according to the content described in S201-S209. Since this process is similar to the specific process described in S201-S209, details are not described here.

[0090] The following will be described in detail Figure 4 The following will be described in detail

[0091] S201: It is determined whether there is an unmatched string without written difference data before the matching string. If there is an unmatched string without written difference data, S202 is performed first, and then S203 is performed. If there is no unmatched string without written difference data, S203 is directly performed.

[0092] In this step, the unmatched string refers to a to-be-compressed string that does not match any string in the second string set.

[0093] S202: The unmatched string and the length of the unmatched string are recorded in the difference data.

[0094] In a specific implementation, the device writes the unmatched string and the length of the unmatched string as an insertion record in the difference data. Specifically, the device can first write the length of the unmatched string in the insertion record, and then determine the data contained in the unmatched string according to the length of the unmatched string and the position of the unmatched string in the to-be-compressed data, so as to write the unmatched string in the insertion record.

[0095] In a specific embodiment, there can be a plurality of unmatched strings before the matched string, which are not written into the differential data, in which case, the total length of the plurality of unmatched strings and the plurality of unmatched strings are written into an insertion record. For example, assuming that there are 2 unmatched strings (i.e., unmatched string 1 and unmatched string 2) before the matched string A1, which are not written into the differential data, then the insertion record written by the device into the differential data includes the total length of the unmatched string 1 and the unmatched string 2, the unmatched string 1, and the unmatched string 2.

[0096] S203: record the offset of the matched string in the reference data and the length of the matched string into the differential data.

[0097] In a specific implementation, the device writes the offset of the matched string in the reference data and the length of the matched string into the differential data as a copy record. The offset of the matched string in the reference data and the length of the matched string are written into the copy record. Optionally, the copy record can also write the actual address of the matched string in the reference data, time information of the copy record, etc., which are not limited herein. In addition, the offset of the matched string in the reference data in the copy record can be the offset of the last byte in the matched string in the reference data, or the offset of the first byte in the matched string in the reference data, etc., which are not limited herein.

[0098] The format of the copy record in the present application is different from the format of the insertion record. Optionally, a flag is added in the record, so that the device can determine whether the record is a copy record or an insertion record according to the flag of the record. For example, if the flag carried in a record is 1, then the record is a copy record, and if the flag carried in a record is 0, then the record is an insertion record. Optionally, a bit in the header information of the record can also be set as an identification bit, the identification bit of the copy record is 1, and the identification bit of the insertion record is 0, etc., which are not limited herein.

[0099] It is worth noting that in a specific implementation, when at least one of the to-be-compressed strings matches any string in the second string set, it can be determined whether there is an unmatched string before the matched to-be-compressed string, which is not written into the differential data. If there is an unmatched string which is not written into the differential data, S202 is executed; if there is no unmatched string which is not written into the differential data, the matched string corresponding to the to-be-compressed string is obtained, and then S203 is executed.

[0100] S204: obtain the first offset and the second offset.

[0101] The first offset is the offset of the last byte in the matching string in the data to be compressed; and the second offset is the offset of the last byte in the partial data in the reference data, or the offset of the last byte in the matching string in the reference data. It can be understood that the second offset is the maximum of the offset of the last byte in the matching string in the reference data and the offset of the last byte in the expanded partial data in the reference data. Specifically, when the offset of the last byte in the matching string in the reference data is greater than the offset of the last byte in the partial data in the reference data, the second offset is the offset of the last byte in the matching string in the reference data; and when the offset of the last byte in the matching string in the reference data is less than or equal to the offset of the last byte in the partial data in the reference data, the second offset is the offset of the last byte in the partial data in the reference data. In this application, the offset can be understood as follows: assuming that the offset of the last byte in the matching string in the data to be compressed is 100, it means that, in the data to be compressed, the last byte in the matching string is the 100th byte of the data to be compressed, relative to the first byte of the data to be compressed.

[0102] S205: determining whether the byte corresponding to the first offset is the last byte in the data to be compressed. If the byte corresponding to the first offset is the last byte in the data to be compressed, S206 is performed; and if the byte corresponding to the first offset is not the last byte in the data to be compressed, S207 is performed.

[0103] The byte corresponding to the first offset is the last byte in the matching string.

[0104] S206: determining whether the data amount of the unmatching string and its length in the difference data is greater than a first threshold. If the data amount of the unmatching string and its length in the difference data is greater than the first threshold, the data to be compressed is compressed again; and if the data amount of the unmatching string and its length in the difference data is less than or equal to the first threshold, the difference data is output.

[0105] The first threshold can be pre-set by a user, or can be dynamically adjusted by the device according to the similarity between the reference data and the data to be compressed, and the like, which is not limited in this application. When the first threshold is a value dynamically adjusted by the device according to the similarity between the reference data and the data to be compressed, the smaller the first threshold is set to be, the higher the similarity between the reference data and the data to be compressed is; and the larger the first threshold is set to be, the lower the similarity between the reference data and the data to be compressed is.

[0106] In the specific embodiment of the present application, the specific process that the device re-compresses the to-be-compressed data is as follows: when the device compresses the to-be-compressed data for the first time, a plurality of to-be-compressed strings have been segmented from the to-be-compressed data, and a second string set has been segmented from the reference data. Therefore, when the device re-compresses the to-be-compressed data, the above plurality of to-be-compressed strings are compared in the second string set first, and subsequent operations are performed according to whether a matched string is obtained. This process is similar to S201-S209, and will not be described in detail here.

[0107] For example, the plurality of to-be-compressed strings segmented from the to-be-compressed data include strings A1, A2, A3, and A4, and the second string set includes strings B1, B2,..., B m+l . Therefore, in the process of re-compressing the to-be-compressed data, the device first compares string A1 in the second string set, so as to determine that string A1 does not match any string in the second string set. Then, the device compares string A2 in the second string set, determines that string A2 matches string B m+l . Therefore, the matched string corresponding to string A2 is obtained, the length of string A1 is recorded, and string A1 is recorded into the delta data. The offset of the matched string corresponding to string A2 in the reference data and the length of the matched string corresponding to string A2 are recorded into the delta data. Next, the device segments the to-be-compressed data starting from the matched string corresponding to string A2, compares the newly segmented to-be-compressed string in the second string set, and performs different operations according to whether a matched string is obtained. This process is repeated until the device completes the compression of data block A.

[0108] It should be noted that in the case where the data amount of the unmatched string and its length in the delta data is greater than the first threshold, the purpose of re-compressing the to-be-compressed data by the device is to ensure that in the case where a large amount of data at the head of the to-be-compressed data is the same as a large amount of data at the tail of the reference data, the device can still compress the large amount of data at the head of the to-be-compressed data according to the reference data. It should also be noted that in the process of compressing the to-be-compressed data last time, the device has segmented the reference data from the head to the tail, and has established a complete fingerprint set. Therefore, in the process of re-compressing the to-be-compressed data, the reference data does not need to be segmented again, and the string at the head of the to-be-compressed data can be matched with the string at the tail of the reference data, thereby solving the problem of poor compression effect caused by the large amount of data at the head of the to-be-compressed data being the same as the large amount of data at the tail of the reference data.

[0109] S207: Determine whether a preset condition is met. If the preset condition is met, perform S208; if the preset condition is not met, perform S209.

[0110] The preset condition includes that a ratio of the first offset and the second offset is greater than or equal to a second threshold, and the second offset corresponds to a last byte in the reference data. The second threshold can be preset by a user or dynamically adjusted by the device according to a data amount of the data to be compressed and a data amount of the reference data, which is not limited here.

[0111] S208: Continue to split the data to be compressed to obtain a target string to be compressed, and expand the expanded partial data again by a fixed step to obtain again expanded partial data, and split the again expanded partial data to obtain a third string set, and compare the target string to be compressed in the third string set.

[0112] In a specific implementation, the device starts from the byte corresponding to the first offset to continue to split the data to be compressed, thereby obtaining a new string to be compressed, that is, a target string to be compressed. The device starts from the byte corresponding to the second offset to expand the expanded partial data again by a fixed step, and split the again expanded partial data to obtain m+l+m strings, thereby obtaining a third string set, wherein the third string set includes strings B1,…,B m+l ,…,B m+l+m .

[0113] In a specific embodiment, the fixed step is less than or equal to a total length of the m strings (strings B m+l+1 ,B m+l+2 ,…,B m+l+m ). The device can expand the partial data to obtain the third string set in the following manner: in step S103, the device has split the partial data into strings B1,B2,…,B m+l Therefore, the device only needs to continue to split the reference data from the byte corresponding to the second offset until m strings (strings B m+l+1 ,B m+l+2 ,…,B m+l+m ) are split, thereby obtaining the third string set. It can be understood that if the amount of data that is not split in the reference data is small, that is, the amount of data that is not split in the reference data is insufficient for the device to split m strings, the fixed step will be less than the total length of the m strings (strings B m+l+1 ,B m+l+2 ,…,B m+l+m ).

[0114] In a specific embodiment, the second threshold is less than 1, thereby ensuring that enough strings are split from the reference data to improve the probability of successful matching of the string to be compressed.

[0115] In a specific embodiment, after obtaining the target to-be-compressed string, the device compares the target to-be-compressed string in the third string set. If a matching string is obtained, the device performs subsequent operations according to the content described with reference to S201-S209; if a matching string is not obtained, the device performs subsequent operations according to the content described with reference to steps S301-S307, until the compression of the to-be-compressed data is completed.

[0116] S209: Continue to split the to-be-compressed data to obtain a target to-be-compressed string, and compare the target to-be-compressed string in the second string set.

[0117] In a specific embodiment, after obtaining the target to-be-compressed string, the device compares the target to-be-compressed string in the third string set. If a matching string is obtained, the device performs subsequent operations according to the content described with reference to S201-S209; if a matching string is not obtained, the device performs subsequent operations according to the content described with reference to steps S301-S307, until the compression of the to-be-compressed data is completed.

[0118] In a specific embodiment, after obtaining the target to-be-compressed string, the device compares the target to-be-compressed string in the second string set. If a matching string is obtained, the device performs subsequent operations according to the content described with reference to S201-S209; if a matching string is not obtained, the device performs subsequent operations according to the content described with reference to steps S301-S307; until the compression of the to-be-compressed data is completed.

[0119] The specific process of how the device operates subsequently in the case where a matching string is not obtained in the step of comparing at least one to-be-compressed string in the second string set in the foregoing step S103 will be explained in detail below, which includes but is not limited to the following steps: Figure 5 The specific process of how the device operates subsequently in the case where a matching string is not obtained in the step of comparing at least one to-be-compressed string in the second string set in the foregoing step S103 will be explained in detail below, which includes but is not limited to the following steps:

[0120] S301: Obtain a first offset and a second offset.

[0121] In this case, the first offset is the offset of the last byte in the at least one to-be-compressed string in the to-be-compressed data, and the second offset is the offset of the last byte in the expanded partial data in the to-be-compressed data. Specifically, the first offset is the offset of the last byte in the string A n in the to-be-compressed data, and the second offset is the offset of the last byte in the string B m+l in the to-be-compressed data.

[0122] S302: Determine whether the byte corresponding to the first offset is the last byte in the to-be-compressed data. If the byte corresponding to the first offset is the last byte in the to-be-compressed data, perform S303-S304; if the byte corresponding to the first offset is not the last byte in the to-be-compressed data, perform S305.

[0123] wherein the first offset corresponds to an offset of the last byte in the at least one to-be-compressed string in the to-be-compressed data.

[0124] S303, record the unmatched string and the length of the unmatched string into the difference data.

[0125] S304: determine whether the data amount of the unmatched string and the length of the unmatched string in the difference data is greater than a first threshold value. If the data amount of the unmatched string and the length of the unmatched string in the difference data is greater than the first threshold value, re-compress the to-be-compressed data; if the data amount of the unmatched string and the length of the unmatched string in the difference data is less than or equal to the first threshold value, output the difference data.

[0126] S305: determine whether a preset condition is met. If the preset condition is met, execute S306; if the preset condition is not met, execute S307.

[0127] wherein a ratio of the first offset and the second offset is greater than or equal to a second threshold value, the second offset corresponds to a byte other than the last byte in the reference data, and the second offset corresponds to an offset of the last byte in the expanded partial data in the to-be-compressed data.

[0128] S306: continue to split the to-be-compressed data to obtain a target to-be-compressed string, expand the partial data by a fixed step, split the expanded partial data to obtain a third string set, and compare the target to-be-compressed string in the third string set.

[0129] S307: continue to split the to-be-compressed data to obtain a target to-be-compressed string, and compare the target to-be-compressed string in the second string set.

[0130] For the sake of simplicity, the above embodiments do not introduce steps S303-S307 in detail. Specifically, the specific implementation process of step S303 is similar to that of step S202, and the specific implementation process of steps S304-S307 is similar to that of steps S206-S209, which will not be described here.

[0131] It should be noted that the above embodiments are described in combination with Figure 4 , Figure 5and its related description (S201-S209, S301-S307), it is not difficult to understand that "obtaining at least one to-be-compressed string from the to-be-compressed data" in S102, wherein the at least one to-be-compressed string can be obtained as follows: when the to-be-compressed data is first divided, a string A1 can be obtained, if the string A1 matches any string in the first string set, S201-S209 is executed, if the string A1 does not match any string in the first string set, S301-S307 is executed, thereby obtaining the at least one to-be-compressed string.

[0132] In the above data compression method, the device divides part of the data in the reference data each time, and then compares the at least one to-be-compressed string divided from the to-be-compressed data with the strings divided from the reference data by using the fingerprints of the strings. When the at least one to-be-compressed string does not match any string divided from the reference data, the device will expand part of the data, and then compare the at least one to-be-compressed string with the newly divided strings from the reference data, until the compression of the to-be-compressed data is completed. It can be seen that, since the device does not need to divide the reference data directly in the above method, the compression of the to-be-compressed data can also be realized, that is, the device does not need to establish a complete fingerprint set corresponding to the reference data, and the compression of the to-be-compressed data can also be realized. Therefore, the above method reduces the operations of calculating the fingerprints of the strings divided from the reference data, determining whether the strings match by comparing the fingerprints of the strings, and the like, thereby greatly reducing the time cost of compressing the to-be-compressed data, and improving the speed of compressing the to-be-compressed data.

[0133] The data compression method provided in the present application can be used to compress the to-be-compressed data, thereby obtaining the differential data. The process of recovering the to-be-compressed data according to the differential data and the reference data will be introduced below.

[0134] First of all, it needs to be pointed out that the differential data obtained according to the foregoing content includes copy records and / or insertion records, and the number of copy records can be one or more, and the number of insertion records can be one or more, which are not limited here. In each copy record, the offset of the matching string in the reference data and the length of the matching string are recorded. In each insertion record, the length of the unmatched string and the unmatched string are recorded.

[0135] In the process of restoring the data to be compressed, the device reads the records in the differential data in sequence. If the read record is a copy record, the device finds the matching string in the reference data according to the position of the matching string in the reference data and the length of the matching string in the copy record, and copies the matching string from the reference data to the output stream. If the read record is an insertion record, the device directly writes the unmatched string from the differential data to the output stream according to the length of the unmatched string and the unmatched string in the insertion record. After the device reads all the records in the differential data, the obtained output stream is the data to be compressed.

[0136] In order to more clearly illustrate the technical solutions provided in the present application, the method provided in the present application is described below through an example (as shown in Figure 6 ).

[0137] Step 21: Selecting part of the data in the reference data, and splitting the part of the data to obtain a first string set (including strings B1, B2, …, B m ); and splitting the data to be compressed from the first byte in the data to be compressed to obtain a string A1.

[0138] Step 22: Comparing the string A1 in the first string set, and the string A1 does not match all the strings in the first string set, at this time the device will continue to split the data to be compressed to obtain a string A2.

[0139] Step 23: Comparing the string A2 in the first string set, and the string A2 does not match all the strings in the first string set, at this time the device will continue to split the data to be compressed to obtain a string A3.

[0140] Step 24: Comparing the string A3 in the first string set, and the string A3 does not match all the strings in the first string set, and satisfies the splitting condition, at this time the device will expand the part of the data by a preset step length, thereby obtaining a second string set (including strings B1, B2, …, B 3×m ).

[0141] In this step, it is assumed that the first preset number in the splitting condition is 3. The preset step length is 2*m strings (i.e. the total length of the strings B m+1 , B m+2 , …, B 3×m) .

[0142] Step 25: The device compares string A1 with the second string set, and string A1 does not match any of the strings in the second string set. At this time, the device compares string A2 with the second string set, and string A2 matches one of the strings in the second string set.

[0143] Step 26: The device writes Insert Record 1 to the differential data. Insert Record 1 contains the length of string A1 and the string A1.

[0144] Step 27: The device obtains the corresponding matching string A2′ of string A2 and writes it to the differential data as copy record 1. Copy record 1 includes the offset of the matching string A2′ in the base data and the length of the matching string A2′.

[0145] Step 28: The device obtains offset 1 and offset 2, and determines that the ratio of offset 1 to offset 2 is less than the second threshold. At this time, the device will continue to split the data to be compressed from the byte corresponding to offset 1 to obtain the string A4.

[0146] In this step, offset 1 is the offset of the last byte of the matched string A2' in the data to be compressed, and offset 2 is the offset of string B. 3×m The position of the last byte in the reference data.

[0147] Step 29: The device compares string A4 with the second string set, and string A4 does not match any of the strings in the second string set.

[0148] Step 210: The device acquires offsets 3 and 4, and determines that the ratio of offset 3 to offset 4 is greater than the second threshold and the byte corresponding to offset 4 is not the last byte in the baseline data. At this point, the device expands part of the data again with a fixed step size, thereby obtaining the third string set (including strings B1, B2, ..., B). 4×m Continue to segment the data to be compressed to obtain the string A5.

[0149] In this step, offset 3 is the offset of the last byte of string A4 within the data to be compressed, and offset 4 is the offset of string B. 3×m The position of the last byte in the reference data. A fixed step size of m strings (i.e., string B) 3×m+1 B 3×m+2 ,…,B 4×m The total length of ).

[0150] Step 211: The device compares string A5 in the third string set and determines that string A5 matches a string in the third string set.

[0151] Step 212: The device writes Insert Record 2 to the differential data. Insert Record 2 contains the length of string A4 and the string A4.

[0152] Step 213: The device obtains the matching string A5′ corresponding to string A5 and writes it to the differential data as copy record 2. Copy record 2 includes the offset of the matching string A5′ in the base data and the length of the matching string A5′.

[0153] Step 214: The device obtains offset 5, thereby determining that the byte corresponding to offset 5 is the last byte in the data to be compressed, and the amount of data to be inserted in the differential data is less than the first threshold, then the device outputs the differential data.

[0154] It can be seen that the above differential data includes Insert Record 1, Copy Record 1, Insert Record 2, and Copy Record 2 in sequence. Therefore, during the decompression process, (1) the device first reads Insert Record 1 in the differential data, and then writes the string A1 recorded in Insert Record 1 directly into the output stream. (2) the device reads Copy Record 1 in the differential data, and finds the matching string A2′ in the reference data according to the offset of the matching string A2′ recorded in Copy Record 1 in the reference data and the length of the matching string A2′, and then copies the matching string A2′ from the reference data to the position after string A1 in the output stream. (3) the device reads Insert Record 2 in the differential data, and then writes the string A4 recorded in Insert Record 2 directly into the position after the matching string A2′ in the output stream. (4) the device reads Copy Record 2 in the differential data, and finds the matching string A5′ in the reference data according to the offset of the matching string A5′ recorded in Copy Record 2 in the reference data and the length of the matching string A5′, and copies the matching string A5′ from the reference data to the position after string A4 in the output stream. At this point, the output stream obtained is the data to be compressed mentioned above.

[0155] Combination Figures 3-6 Following the description of related information, the following section introduces the relevant data compression apparatus involved in the embodiments of this application. For example... Figure 7 As shown, Figure 7 A schematic diagram of the structure of a data compression device provided in this application is shown. Figure 7 The data compression device 100 shown includes: a segmentation unit 110, a matching unit 120, a recording unit 130, and a recompression unit 140, wherein,

[0156] The segmentation unit 110 is used to select a portion of the data in the reference data and segment the portion of the data to obtain a first string set.

[0157] The splitting unit 110 is further configured to acquire at least one to-be-compressed string from the to-be-compressed data.

[0158] The matching unit 120 is configured to match the at least one to-be-compressed string in the first string set.

[0159] The splitting unit 110 is further configured to, in a case where no matched string is acquired, expand the partial data in the reference data by a preset step length, and split the expanded partial data to obtain a second string set.

[0160] The matching unit 120 is further configured to match the at least one to-be-compressed string in the second string set.

[0161] The recording unit 130 is configured to, in a case where a matched string is acquired, record an offset and a length of the matched string in the reference data into the difference data.

[0162] In a specific embodiment, after the matching unit 120 matches the at least one to-be-compressed string in the second string set, the recording unit 130 is further configured to, in a case where no matched string is acquired, record the unmatched string and a length of the unmatched string into the difference data.

[0163] In a specific embodiment, after the matching unit 120 matches the at least one to-be-compressed string in the first string set, the recording unit 130 is further configured to, in a case where a matched string is acquired, record an offset and a length of the matched string in the reference data into the difference data.

[0164] In a specific embodiment, the re-compression unit 140 is configured to, in a case where a last byte in the matched string is a last byte of the to-be-compressed data, calculate a data amount of the unmatched string and a length of the unmatched string in the difference data, and re-compress the to-be-compressed data in a case where the data amount is greater than a first threshold.

[0165] In a specific embodiment, the splitting unit 110 is further configured to determine whether a splitting condition is met, and expand the partial data by a preset step length in a case where the splitting condition is met, wherein the splitting condition comprises that a number of the at least one to-be-matched string is less than or equal to a first preset number, and a number of strings in the first string set is less than or equal to a second preset number.

[0166] In a specific embodiment, the splitting unit 110 is further configured to obtain a first offset and a second offset, wherein the first offset is an offset of a last byte in the matching string in the data to be compressed, or an offset of a last byte in the at least one string to be compressed in the data to be compressed; the second offset is an offset of the last byte in the matching string in the reference data, or an offset of a last byte in the partial data in the reference data; and the splitting unit 110 is further configured to split the target string to be compressed from the data to be compressed if a ratio of the first offset and the second offset is less than a second threshold; the matching unit 120 is further configured to compare the target string to be compressed in the second string set; and the recording unit 130 is further configured to record, if the matching string is obtained, an offset and a length of the matching string in the reference data into the difference data.

[0167] In a specific embodiment, the splitting unit 110 is further configured to split the target string to be compressed from the data to be compressed if a ratio of the first offset and the second offset is greater than or equal to the second threshold, and to expand the partial data by a fixed step and split the expanded partial data to obtain a third string set; the matching unit 120 is further configured to compare the target string to be compressed in the third string set; and the recording unit 130 is further configured to record, if the matching string is obtained, an offset and a length of the matching string in the reference data into the difference data.

[0168] For example, after the matching unit 120 compares the at least one to-be-compressed string in the second string set, if no matching string is found, the recording unit 130 records the non-matching string and the length of the non-matching string in the difference data. Then, the splitting unit 110 obtains a first offset and a second offset, wherein the first offset is the offset of the last byte in the at least one to-be-compressed string in the to-be-compressed data, and the second offset is the offset of the last byte in the expanded partial data in the to-be-compressed data. Then, the splitting unit 110 determines whether the byte corresponding to the first offset is the last byte in the to-be-compressed data. If the splitting unit 110 determines that the byte corresponding to the first offset is the last byte in the to-be-compressed data, the recording unit 130 records the non-matching string and the length of the non-matching string in the difference data, the re-compression unit 140 determines whether the data amount of the non-matching string and the length of the non-matching string in the difference data is greater than the first threshold, if the data amount of the non-matching string and the length of the non-matching string in the difference data is greater than the first threshold, the to-be-compressed data is re-compressed; if the data amount of the non-matching string and the length of the non-matching string in the difference data is less than or equal to the first threshold, the difference data is output. If the splitting unit 110 determines that the byte corresponding to the first offset is not the last byte in the to-be-compressed data, it is determined whether the preset condition is met, if the preset condition is met, the to-be-compressed data is continuously split to obtain a target to-be-compressed string, the partial data is expanded by a fixed step, the expanded partial data is split to obtain a third string set, and the target to-be-compressed string is compared in the third string set; if the preset condition is not met, the to-be-compressed data is continuously split to obtain a target to-be-compressed string, and the target to-be-compressed string is compared in the second string set.

[0169] For example, after the matching unit 120 compares the at least one to-be-compressed string with the second string set, in the case that a matched string is obtained, the recording unit 130 records the offset and length of the matched string in the reference data into the difference data. Then, the splitting unit 110 determines whether there is an unmatched string that is not written into the difference data before the matched string, if there is an unmatched string that is not written into the difference data, the recording unit 130 first records the unmatched string and the length of the unmatched string into the difference data, and then records the offset of the matched string in the reference data and the length of the matched string into the difference data; if there is no unmatched string that is not written into the difference data, the recording unit 130 directly records the offset of the matched string in the reference data and the length of the matched string into the difference data. Then, the splitting unit 110 obtains a first offset and a second offset, wherein the first offset is the offset of the last byte in the matched string in the to-be-compressed data; and the second offset is the offset of the last byte in the partial data in the reference data, or the offset of the last byte in the matched string in the reference data. Then, the splitting unit 110 determines whether the byte corresponding to the first offset is the last byte in the to-be-compressed data, if the splitting unit 110 determines that the byte corresponding to the first offset is the last byte in the to-be-compressed data, it is determined whether the data amount of the unmatched string and the length of the unmatched string in the difference data is greater than a first threshold, if the data amount of the unmatched string and the length of the unmatched string in the difference data is greater than the first threshold, the to-be-compressed data is compressed again, if the data amount of the unmatched string and the length of the unmatched string in the difference data is less than or equal to the first threshold, the difference data is output. If the splitting unit 110 determines that the byte corresponding to the first offset is not the last byte in the to-be-compressed data, it is determined whether a preset condition is met, if the preset condition is met, the to-be-compressed data is continuously split to obtain a target to-be-compressed string, the partial data is enlarged by a fixed step, the enlarged partial data is split to obtain a third string set, and the target to-be-compressed string is compared with the third string set; if the preset condition is not met, the to-be-compressed data is continuously split to obtain a target to-be-compressed string, and the target to-be-compressed string is compared with the second string set.

[0170] In a specific embodiment, the matched string includes the to-be-compressed string and / or the enlarged to-be-compressed string, the enlarged to-be-compressed string is obtained by enlarging the matched to-be-compressed string, so that the reference data and the to-be-compressed data both include the enlarged to-be-compressed string.

[0171] The data compression apparatus 100 in the embodiments of the present application is only used as an example for dividing the above-mentioned functions when compressing the data to be compressed. In actual application, the above-mentioned functions can be completed by different function modules according to the needs, that is, the internal structure of the data compression apparatus 100 is divided into different function modules to complete all or part of the above-mentioned functions. In addition, the data compression apparatus 100 provided by the above-mentioned embodiments and the above-mentioned method embodiments belong to the same concept, and the specific implementation process and the related explanations of the matching string, the non-matching string and other concepts are described in the above-mentioned method embodiments, which will not be repeated here.

[0172] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of an electronic device provided by the present application. Figure 8 The electronic device 200 shown can be the data compression apparatus 100 in the foregoing content, or a hardware device running the data compression apparatus 100. The electronic device 200 includes a processor 210, a communication interface 220 and a memory 230. The electronic device includes that the processor 210, the communication interface 220 and the memory 230 are coupled through a bus 240.

[0173] The processor 210 can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), a transistor logic device, a hardware component or any combination thereof. The processor 210 can implement or execute various exemplary methods described in combination with the disclosure. Specifically, the processor 210 reads the program code stored in the memory 230 and cooperates with the communication interface 220 to execute part or all of the steps of S101-S104, S201-S209 and S301-S307.

[0174] The communication interface 220 can be a wired interface or a wireless interface, used for communication with other modules or devices. The wired interface can be an Ethernet interface, a controller area network interface, a local interconnect network (LIN) and a FlexRay interface, and the wireless interface can be a cellular network interface or a wireless local area network interface, etc.

[0175] Memory 230 may include volatile memory, such as random access memory (RAM); memory 230 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD); memory 230 may also include combinations of the above types of memory. Memory 230 may store program code and program data. The program code is generated by... Figure 7 The data compression device shown is composed of codes for some or all of its units, such as the code for the segmentation unit 110, the code for the matching unit 120, the code for the recording unit 130, and the code for the recompression unit 140. The program data is composed of... Figure 7 The data compression device shown generates data during program execution, such as the string to be compressed, the first set of strings, the matching string, copied records, inserted records, etc.

[0176] Bus 240 can be a controller area network (CAN) or other implementation of an internal bus. Bus 240 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0177] The data compression device in this application embodiment is used to execute the method executed in the above method embodiment. It belongs to the same concept as the above method embodiment, and its specific implementation process is detailed in the above method embodiment, which will not be repeated here.

[0178] This application also provides a computer-readable storage medium storing a computer program, which is executed by hardware (e.g., a processor) to implement some or all of the steps in the data compression method (S101-S104, S201-S209, S301-S307) provided in this application.

[0179] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The above computer program product includes one or more computer instructions. When loaded and executed on a computer, all or part of the above computer program instructions generate the processes or functions described in the embodiments of the present application. The above computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The above computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the above computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The above computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media. The above available medium can be a magnetic medium (such as a floppy disk, a storage disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD), etc. In the above embodiments, the description of each embodiment is focused on each embodiment, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0180] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can also be implemented by other manners. For example, the apparatus embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the indirect coupling or direct coupling or communication connection between the units shown or discussed can be through some interface, apparatus or unit, and can be electrical or other forms.

[0181] The units described as separate components above can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the scheme of the embodiments of the present application.

[0182] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software functional unit.

[0183] The integrated unit described above, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium, for example, can include: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various program code storage media.

[0184] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data compression method, characterized in that, The method includes: Select a portion of the baseline data, and segment the portion of the data to obtain a first string set; Multiple strings to be compressed are obtained from the data to be compressed, and the multiple strings to be compressed are compared in the first string set; If each of the plurality of strings to be compressed does not match the strings in the first string set, the partial data is expanded in the reference data by a preset step size, and the expanded partial data is segmented to obtain a second string set. The plurality of strings to be compressed are compared in the second string set. If a matching string is obtained, the offset and length of the matching string in the reference data are recorded in the difference data.

2. The method according to claim 1, characterized in that, The method further includes: The multiple strings to be compressed are compared in the second string set. If no matching string is found, the unmatched string and its length are recorded in the difference data.

3. The method according to claim 1, characterized in that, After comparing the plurality of strings to be compressed within the first string set, the method further includes: If a matching string is obtained, the offset and length of the matching string in the baseline data are recorded in the difference data.

4. The method according to claim 1, characterized in that, After recording the offset and length of the matching string in the baseline data into the difference data, the method further includes: If the last byte in the matched string is the last byte of the data to be compressed, calculate the amount of data in the difference data that contains unmatched strings and the length of the unmatched strings. If the amount of data exceeds the first threshold, the data to be compressed is recompressed.

5. The method according to any one of claims 1-4, characterized in that, Before expanding the partial data by a preset step size, the method further includes: Determine whether the segmentation conditions are met. If the segmentation conditions are met, expand the partial data by a preset step size. The segmentation conditions include: the number of the plurality of strings to be compressed is less than or equal to a first preset number, and the number of strings in the first string set is less than or equal to a second preset number.

6. The method according to claim 5, characterized in that, After comparing the plurality of strings to be compressed within the second string set, the method further includes: Obtain a first offset and a second offset, wherein the first offset is the offset of the last byte in the matched string in the data to be compressed, or the offset of the last byte in the plurality of strings to be compressed in the data to be compressed; the second offset is the offset of the last byte in the matched string in the reference data, or the offset of the last byte in the partial data in the reference data; If the ratio of the first offset to the second offset is less than the second threshold, the target string to be compressed is segmented from the data to be compressed. The target string to be compressed is compared with the second string set. If a matching string is found, the offset and length of the matching string in the baseline data are recorded in the difference data.

7. The method according to claim 6, characterized in that, The method further includes: If the ratio of the first offset to the second offset is greater than or equal to the second threshold, the target string to be compressed is segmented from the data to be compressed, the portion of data is expanded with a fixed step size, and the expanded portion of data is segmented to obtain a third string set. The target string to be compressed is compared with the third string set. If a matching string is found, the offset and length of the matching string in the baseline data are recorded in the difference data.

8. The method according to claim 1, characterized in that, The matching string includes the matched string to be compressed and / or the amplified string to be compressed. The amplified string to be compressed is obtained by amplifying the matched string to be compressed, so that both the base data and the data to be compressed include the amplified string to be compressed.

9. A data compression device, characterized in that, The device includes a segmentation unit, a matching unit, and a recording unit. The segmentation unit is used to select a portion of data from the reference data and segment the portion of data to obtain a first string set. The segmentation unit is also used to obtain multiple strings to be compressed from the data to be compressed; The matching unit is used to compare the plurality of strings to be compressed in the first string set; The segmentation unit is further configured to, when each of the plurality of strings to be compressed does not match the strings in the first string set, expand the partial data in the reference data by a preset step size, and segment the expanded partial data to obtain a second string set. The matching unit is used to compare the plurality of strings to be compressed in the second string set; The recording unit is used to record the offset and length of the matching string in the baseline data into the difference data when a matching string is obtained.

10. The apparatus according to claim 9, characterized in that, After the matching unit compares the plurality of strings to be compressed within the second string set, The recording unit is also used to record the unmatched string and the length of the unmatched string into the difference data when no matching string is obtained.

11. The apparatus according to claim 9, characterized in that, After the matching unit compares the plurality of strings to be compressed within the first string set... The recording unit is also used to record, when a matching string is obtained, the offset and length of the matching string in the reference data into the difference data.

12. The apparatus according to claim 9, characterized in that, The apparatus further includes a recompression unit, the recompression unit being used for: If the last byte of the matched string is the last byte of the data to be compressed, calculate the amount of data in the differential data that contains unmatched strings and the length of the unmatched strings. If the amount of data exceeds the first threshold, the data to be compressed is recompressed.

13. The apparatus according to any one of claims 9-12, The segmentation unit is further configured to determine whether the segmentation conditions are met. If the segmentation conditions are met, the portion of data is expanded by a preset step size. The segmentation conditions include: the number of the plurality of strings to be compressed is less than or equal to a first preset number, and the number of strings in the first string set is less than or equal to a second preset number.

14. The apparatus according to claim 13, characterized in that, The segmentation unit is further configured to obtain a first offset and a second offset, wherein the first offset is the offset of the last byte in the matched string in the data to be compressed, or the offset of the last byte in the plurality of strings to be compressed in the data to be compressed; the second offset is the offset of the last byte in the matched string in the reference data, or the offset of the last byte in the partial data in the reference data. The segmentation unit is further configured to segment the target string to be compressed from the data to be compressed when the ratio of the first offset to the second offset is less than the second threshold. The matching unit is also used to compare the target string to be compressed in the second string set; The recording unit is also used to record the offset and length of the matching string in the reference data into the difference data when a matching string is obtained.

15. The apparatus according to claim 14, characterized in that, The segmentation unit is further configured to, when the ratio of the first offset to the second offset is greater than or equal to the second threshold, segment the target string to be compressed from the data to be compressed, expand the portion of data with a fixed step size, and segment the expanded portion of data to obtain a third string set. The matching unit is also used to compare the target string to be compressed in the third string set; The recording unit is also used to record the offset and length of the matching string in the reference data into the difference data when a matching string is obtained.

16. The apparatus according to claim 9, characterized in that, The matching string includes the matched string to be compressed and / or the amplified string to be compressed. The amplified string to be compressed is obtained by amplifying the matched string to be compressed, so that both the base data and the data to be compressed include the amplified string to be compressed.

17. An electronic device, the electronic device comprising a processor and a memory, wherein, The memory stores code, and the processor executes the code in the memory to implement the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 8.

Images