A data compression method and device, electronic equipment and storage medium

Abstract

The application discloses a data compression method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring to-be-compressed data; determining a current data unit in the to-be-compressed data; the current data unit comprises a first preset number of bytes; taking each byte in the current data unit as a first byte, extracting a plurality of to-be-processed data units in the to-be-compressed data; each to-be-processed data unit comprises a second preset number of bytes; performing parallel matching on the plurality of to-be-processed data units and previous data by using a plurality of calculation modules; determining a target to-be-processed data unit matched to a corresponding matching data unit, taking the position of the first byte in the matching data unit in the to-be-compressed data as a matching position, taking the length of the matching data unit as a matching length, and replacing the target to-be-processed data unit in the to-be-compressed data with the matching position and the matching length. The application performs parallel matching on repeated data, and improves the compression performance of the LZ77 algorithm.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data compression method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Data compression can potentially reduce data storage space and increase the logical capacity of storage devices, thereby reducing data storage and transmission costs, making it a highly attractive technology. Data compression is a computationally intensive operation, requiring significant host CPU (central processing unit) computing resources. Implementing data compression hardware circuitry within solid-state drives (SSDs), using a computationally oriented storage architecture to support data compression, has been a technological trend in recent years.

[0003] The LZ77 algorithm achieves compression by replacing recurring data fragments with references to single copies of earlier data fragments in the uncompressed data stream. A matched fragment is represented by a pair of numbers called "length-distance". In related techniques, the LZ77 algorithm performs matching at each position of the input data sequentially, resulting in slow execution speed and low compression performance.

[0004] Therefore, how to improve the compression performance of the LZ77 algorithm is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data compression method, apparatus, electronic device, and computer-readable storage medium, which improves the compression performance of the LZ77 algorithm.

[0006] To achieve the above objectives, this application provides a data compression method, comprising:

[0007] Obtain the data to be compressed;

[0008] The current data unit is determined in the data to be compressed; wherein the current data unit includes a first preset number of bytes;

[0009] Using each byte in the current data unit as the first byte, multiple data units to be processed are extracted from the data to be compressed; wherein each data unit to be processed includes a second preset number of bytes;

[0010] Multiple computing modules are used in parallel to match multiple data units to be processed with previous data;

[0011] Determine the target data unit to be processed that matches the corresponding matching data unit, take the position of the first byte in the matching data unit in the data to be compressed as the matching position, take the length of the matching data unit as the matching length, and replace the target data unit to be processed in the data to be compressed with the matching position and the matching length.

[0012] Determining the current data unit in the data to be compressed includes:

[0013] Determine the current processing location;

[0014] Using the byte corresponding to the current processing position in the data to be compressed as the first byte, extract the current data unit containing the first preset number of bytes from the data to be compressed;

[0015] Accordingly, after using multiple computing modules to match multiple data units to be processed with the previous data in parallel, the process further includes:

[0016] If there is no target data unit to be processed that matches the corresponding matching data unit, the current processing position is incremented by the first preset number, and the process re-enters the step of extracting the current data unit containing the first preset number of bytes in the data to be compressed, with the byte corresponding to the current processing position in the data to be compressed as the first byte.

[0017] Accordingly, after replacing the target data unit to be processed in the data to be compressed with the matching position and the matching length, the method further includes:

[0018] The current processing position is incremented by the matching length, and the process re-enters the step of extracting the current data unit containing the first preset number of bytes from the data to be compressed, using the byte corresponding to the current processing position in the data to be compressed as the first byte.

[0019] The step of using multiple computing modules to match multiple data units to be processed with previous data in parallel includes:

[0020] Each computing module calculates the target hash value of the corresponding data unit to be processed. Using the target hash value as an index, the corresponding target hash entry is determined in a hash table. Candidate matching positions are determined within the target hash entries. First data content is read starting from the candidate matching position in the data to be compressed, and second data content is read starting from the data unit to be processed in the data to be compressed. The first data content and the second data content are then matched. The hash table stores the correspondence between the hash values ​​of the data content and their positions in the data to be compressed.

[0021] Accordingly, determining the target data unit to be processed that matches the corresponding matching data unit includes:

[0022] If the first data content and the second data content match successfully, the data unit to be processed contained in the second data content is determined as the target data unit to be processed, and the first data content is determined as the matching data unit.

[0023] After determining the target data unit to be processed that matches the corresponding matching data unit, the method further includes:

[0024] The hash entry corresponding to the hash value of the target data unit to be processed in the hash table is updated based on the position of the first byte in the data to be compressed.

[0025] Accordingly, after using multiple computing modules to match multiple data units to be processed with the previous data in parallel, the process further includes:

[0026] If no target data unit to be processed is found that matches the corresponding matching data unit, then the hash entries corresponding to the hash values ​​of the multiple data units to be processed in the hash table are updated based on the position of the first byte in the data to be compressed.

[0027] After determining the target data unit to be processed that matches the corresponding matching data unit, the method further includes:

[0028] If the second data content containing the first target data unit to be processed overlaps with the second data content containing the second target data unit to be processed, then the target data unit that is later in position between the first target data unit to be processed and the second target data unit to be processed will be removed.

[0029] The hash table includes multiple hash entries. Each hash entry is indexed by a first-level hash value calculated based on a first hash algorithm. Each hash entry includes a second preset number of second-level hash values ​​and a second preset number of positions in the data to be compressed. The second-level hash value is a hash value calculated based on a second hash algorithm for the data content. Both the first-level hash value and the second-level hash value are one byte.

[0030] The step of calculating the target hash value of the corresponding data unit to be processed using each computing module, determining the corresponding target hash entry in the hash table using the target hash value as an index, and determining the candidate matching position in the target hash entry includes:

[0031] Each computing module calculates the target first-level hash value of the corresponding data unit to be processed based on the first hash algorithm, and uses the target first-level hash value as an index to determine the corresponding target hash entry in the hash table;

[0032] Each computing module calculates the target secondary hash value of the corresponding data unit to be processed based on the second hash algorithm, and determines the candidate matching position corresponding to the target secondary hash value in the target hash entry.

[0033] To achieve the above objectives, this application provides a data compression apparatus, comprising:

[0034] The acquisition module is used to acquire the data to be compressed;

[0035] A determining module is used to determine the current data unit in the data to be compressed; wherein the current data unit includes a first preset number of bytes;

[0036] An extraction module is used to extract multiple data units to be processed from the data to be compressed, using each byte in the current data unit as the first byte; wherein each data unit to be processed includes a second preset number of bytes;

[0037] Multiple computing modules are used to match multiple data units to be processed with previous data in parallel;

[0038] The replacement module is used to determine the target data unit to be processed that matches the corresponding matching data unit, take the position of the first byte in the matching data unit in the data to be compressed as the matching position, take the length of the matching data unit as the matching length, and replace the target data unit to be processed in the data to be compressed with the matching position and the matching length.

[0039] To achieve the above objectives, this application provides an electronic device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor for executing the computer program to implement the steps of the data compression method described above.

[0042] To achieve the above objectives, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data compression method described above.

[0043] As can be seen from the above scheme, the data compression method provided in this application includes: acquiring data to be compressed; determining a current data unit in the data to be compressed; wherein the current data unit includes a first preset number of bytes; taking each byte in the current data unit as the first byte, extracting multiple data units to be processed in the data to be compressed; wherein each data unit to be processed includes a second preset number of bytes; using multiple computing modules to match the multiple data units to be processed with previous data in parallel; determining a target data unit to be processed that matches the corresponding matching data unit, taking the position of the first byte in the matching data unit in the data to be compressed as the matching position, taking the length of the matching data unit as the matching length, and replacing the target data unit to be processed in the data to be compressed with the matching position and the matching length.

[0044] The data compression method provided in this application utilizes multiple computing modules to perform parallel matching of duplicate data on multiple data units to be processed, thereby significantly improving the compression performance of the LZ77 algorithm. This application also discloses a data compression apparatus, an electronic device, and a computer-readable storage medium, which can achieve the same technical effects.

[0045] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the following detailed description to explain this disclosure, but do not constitute a limitation of this disclosure. In the drawings:

[0047] Figure 1 This is a flowchart of the duplicate data identification technique in the LZ77 algorithm of related technologies;

[0048] Figure 2 This is a schematic diagram of a duplicate data identification technology based on serial computation in related technologies;

[0049] Figure 3 This is a structural diagram of the hash table in the LZ77 algorithm of related technologies;

[0050] Figure 4 This is a flowchart illustrating a data compression method according to an exemplary embodiment;

[0051] Figure 5 A flowchart illustrating another data compression method according to an exemplary embodiment;

[0052] Figure 6 This is a structural diagram of a hash table according to an exemplary embodiment;

[0053] Figure 7 for Figure 5 Detailed flowchart of step S26;

[0054] Figure 8 A flowchart illustrating a duplicate data identification technique based on parallel computing and memory optimization according to an exemplary embodiment;

[0055] Figure 9 This is a schematic diagram illustrating a duplicate data identification technique based on parallel computing according to an exemplary embodiment;

[0056] Figure 10 This is a structural diagram of a data compression apparatus according to an exemplary embodiment;

[0057] Figure 11 This is a structural diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, in the embodiments of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0059] In related technologies, the algorithm flow of duplicate data identification technology in the LZ77 algorithm is as follows: Figure 1 As shown, if the last byte is more than 4 bytes away from the start of the current data unit, then there is at least one 4-byte data unit in the current data stream, and matching can continue; otherwise, matching ends. The matching logic is as follows:

[0060] Step 1: Calculate the 2-byte hash value of the current data unit;

[0061] Step 2: Look up the hash table using the hash value as the index;

[0062] Step 3: If a hash hit occurs, continue to obtain the matching position and matching length; starting from each matching byte, calculate the hash value of its corresponding 4-byte data unit and update the hash table; increment the starting position of the current data unit by the matching length bytes.

[0063] Step 4: If the hash is not found, update the hash table with the current hash value and increment the current data cell position by 1 byte.

[0064] Step 5: Return to step 1 to proceed with the next round of matching.

[0065] In related technologies, the LZ77 algorithm is executed serially, such as... Figure 2 As shown, each time a 4-byte data unit is taken from a starting position i, a hash operation is performed to find a potential match. If the hash matches, the matching position is obtained, and subsequent data comparisons are performed to determine the matching length ML. If position i successfully matches, the starting position is moved to i+ML for the next round of matching; if position i does not match, the next round of matching starts from i+1. It is evident that one of the drawbacks of the LZ77 algorithm in related technologies is its slow serial execution speed, which cannot meet the data compression performance requirements of scenarios such as computationally intensive storage.

[0066] In related technologies, the structure diagram of the hash table in the LZ77 algorithm is as follows: Figure 3 As shown, it typically contains 32,768 entries, with each entry storing two position information entries (pos1 and pos2) totaling 4 bytes. Therefore, the size of the entire hash table is 32,768 * 4 = 131,072 bytes (128KB). It is evident that the second drawback of the LZ77 algorithm in related technologies is the large memory overhead of the hash table, which is unacceptable in memory-scarce scenarios such as hardware implementations.

[0067] Based on this, this application designs a minimal hash table containing only 256 entries to address the problem of high memory overhead, and supports multi-path parallelism to accelerate the identification of duplicate data segments. This application can save on hardware implementation memory overhead, significantly reduce the integrated circuit area of ​​ASIC (Application Specific Integrated Circuit) chips, and support multiple hardware computing units, thereby multiplying the speed of the duplicate data matching algorithm.

[0068] This application discloses a data compression method that improves the compression performance of the LZ77 algorithm.

[0069] See Figure 4 A flowchart illustrating a data compression method according to an exemplary embodiment, such as... Figure 4 As shown, it includes:

[0070] S11: Obtain the data to be compressed;

[0071] S12: Determine the current data unit in the data to be compressed; wherein the current data unit includes a first preset number of bytes;

[0072] The purpose of this embodiment is to compress the data to be compressed using an improved LZ77 algorithm. In specific implementation, the data to be compressed is obtained, and a current data unit containing a first preset number of bytes is determined within it. For example, if the starting position of the current data unit is i, and the first preset number is 4, the current data unit is [i, i+1, i+2, i+3]. Or, if the first preset number is 8, the current data unit is [i, i+1, i+2, i+3, i+4, i+5, i+6, i+7].

[0073] S13: Using each byte in the current data unit as the first byte, extract multiple data units to be processed from the data to be compressed; wherein each data unit to be processed includes a second preset number of bytes;

[0074] In this step, each byte in the current data unit is used as the first byte to extract a data unit containing a second preset number of bytes from the data to be compressed. For example, if the current data unit is [i,i+1,i+2,i+3] and the second preset number is 4, the data units to be processed are [i,i+1,i+2,i+3], [i+1,i+2,i+3,i+4], [i+2,i+3,i+4,i+5], and [i+3,i+4,i+5,i+6]. For another example, the current data unit is [i,i+1,i+2,i+3,i+4,i+5,i+6,i+7], and when the second preset number is 4, the data units to be processed are [i,i+1,i+2,i+3], [i+1,i+2,i+3,i+4], [i+2,i +3,i+4,i+5], [i+3,i+4,i+5,i+6], [i+4,i+5,i+6,i+7], [i+5,i+6,i+7,i+8], [i+6,i+7,i+8,i+9], [i+7,i+8,i+9,i+10].

[0075] S14: Multiple computing modules are used to match multiple data units to be processed with the previous data in parallel;

[0076] S15: Determine the target data unit to be processed that matches the corresponding matching data unit, take the position of the first byte in the matching data unit in the data to be compressed as the matching position, take the length of the matching data unit as the matching length, and replace the target data unit to be processed in the data to be compressed with the matching position and the matching length.

[0077] In practical implementation, multiple computing modules are used to perform parallel matching of duplicate data on multiple data units to be processed, which significantly improves the matching speed in the LZ77 algorithm. If no target data unit to be processed is found to match, the process re-enters S12. If a target data unit to be processed is found to match, the matching position and matching length are determined, and the target data unit to be processed in the data to be compressed is replaced with the matching position and matching length to achieve data compression.

[0078] As a feasible implementation, the method of using multiple computing modules to match multiple data units to be processed with previous data in parallel includes: calculating the target hash value of the corresponding data unit to be processed using each computing module; determining the corresponding target hash entry in a hash table using the target hash value as an index; determining candidate matching positions in the target hash entries; reading first data content starting from the candidate matching position in the data to be compressed; reading second data content starting from the data unit to be processed in the data to be compressed; and matching the first data content and the second data content. The hash table is used to store the correspondence between the hash values ​​of data content and their positions in the data to be compressed. Correspondingly, determining the target data unit to be processed that matches the corresponding matching data unit includes: if the first data content and the second data content match successfully, then determining the data unit to be processed contained in the second data content as the target data unit to be processed, and determining the first data content as the matching data unit.

[0079] In practical implementation, a hash table is used to store the correspondence between the hash value of the data content and its position in the data to be compressed. The hash table includes multiple hash entries, each with an index representing the hash value of the data content and storing the position of that data content in the data to be compressed—specifically, the position of the first byte of that data content within the data to be compressed. After extracting multiple data units to be processed from the data to be compressed, they are input into multiple computing modules. Each computing module performs a duplicate data matching operation on the input data units. Specifically, the process involves: calculating the target hash value of the data unit; using the target hash value as an index to determine the corresponding target hash entry in the hash table; and checking whether the target hash entry stores a valid value. If it does not exist, the match fails, and the hash entry corresponding to the hash value of the data unit in the hash table is updated based on the position of the first byte of the data unit in the data to be compressed—that is, the target hash entry is updated. If a match exists, the valid value is read from the target hash entry as a candidate matching position. The first data content is read starting from the candidate matching position in the data to be compressed, and the second data content is read starting from the data unit to be processed in the data to be compressed. Specifically, the second data content is read starting from the first byte of the data unit to be processed in the data to be compressed. The first and second data contents are matched. If the first and second data contents match successfully, the matching position and matching length are determined. Furthermore, the hash entry corresponding to the hash value of the target data unit to be processed in the hash table is updated based on the position of the first byte in the target data unit to be processed within the data to be compressed.

[0080] For example, if the first byte of the data to be processed is located at position 1000 in the data to be compressed, and the data to be processed is specifically ABCD with a corresponding hash value of 8888, then the 8888th entry in the hash table is searched using 8888 as the index. It is checked whether a valid value is stored there. If not, the match fails, and 1000 is stored in the 8888th entry. If it is, the valid value 600 is read from the 8888th entry, and data content is read from positions 600 and 1000 in the data to be compressed for matching. For example, if the data content read from position 600 is ABCDEF…, and the data content read from position 600 is ABCDEG…, then the matching position is 600, the matching length is 5, and 600 in the 8888th entry is replaced with 1000.

[0081] In a preferred embodiment, after determining the target data unit to be processed that matches the corresponding matching data unit, the method further includes: if the second data content containing the first target data unit to be processed and the second data content containing the second target data unit to be processed overlap, then the target data unit to be processed that is later in position between the first target data unit to be processed and the second target data unit to be processed is removed.

[0082] It should be noted that during the parallel execution of duplicate data matching by multiple computing modules on multiple data units to be processed, multiple computing modules may match the data, that is, identify multiple target data units to be processed that match the corresponding matching data units. In this case, if the second data content containing multiple target data units to be processed overlaps, the target data unit to be processed that is later in position is removed, the matching position and matching length of the other target data units to be processed that have not been removed are determined and replaced, and the hash table is updated based on the position of the other target data units to be processed that have not been removed in the data to be compressed.

[0083] For example, 8 data units to be processed: [i,i+1,i+2,i+3], [i+1,i+2,i+3,i+4], [i+2,i+3,i+4,i+5], [i+3,i+4,i+5,i+6], [i+4,i+5 ,i+6,i+7], [i+5,i+6,i+7,i+8], [i+6,i+7,i+8,i+9], [i+7,i+8,i+9,i+10], respectively recorded as A0, A1, A2, A3, A4, A5, A6, and A7. The matching results are as follows: A0 has a corresponding matching data unit with a matching length of 6. The second data content M1 containing A0 is [i,i+1,i+2,i+3,i+4,i+5]. A4 ​​has a corresponding matching data unit with a matching length of 4. The second data content M2 containing A4 is [i+4,i+5,i+6,i+7]. A6 has a corresponding matching data unit with a matching length of 4. The second data content M3 containing A6 is [i+6,i+7,i+8,i+9]. Since M1 and M2 overlap, A4 is removed. After removing A4, M3 does not overlap with M1. Therefore, A0 and A6 are retained as target data units to be processed.

[0084] The data compression method provided in this application utilizes multiple computing modules to perform parallel matching of duplicate data on multiple data units to be processed, thereby significantly improving the compression performance of the LZ77 algorithm.

[0085] This application discloses a data compression method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0086] See Figure 5 A flowchart illustrating another data compression method according to an exemplary embodiment, such as... Figure 5 As shown, it includes:

[0087] S21: Obtain the data to be compressed;

[0088] S22: Determine the current processing position;

[0089] In this embodiment, the initial current processing position is 0.

[0090] S23: Determine whether the distance between the current processing position and the last byte in the data to be compressed is greater than or equal to a first preset number; if yes, proceed to S24; if no, end the process.

[0091] In practice, it is determined whether the distance between the current processing position and the last byte in the data to be compressed is greater than or equal to a first preset number. If so, proceed to S24; otherwise, end the process.

[0092] S24: Using the byte corresponding to the current processing position in the data to be compressed as the first byte, extract the current data unit containing the first preset number of bytes from the data to be compressed;

[0093] S25: Using each byte in the current data unit as the first byte, extract multiple data units to be processed from the data to be compressed; wherein each data unit to be processed includes a second preset number of bytes;

[0094] S26: Calculate the target hash value of the corresponding data unit to be processed using each computing module, determine the corresponding target hash entry in the hash table using the target hash value as an index, determine the candidate matching position in the target hash entry, read the first data content starting from the candidate matching position in the data to be compressed, read the second data content starting from the data unit to be processed in the data to be compressed, and match the first data content and the second data content; wherein, the hash table is used to store the correspondence between the hash value of the data content and the position in the data to be compressed; if the first data content and the second data content match successfully, then the data unit to be processed contained in the second data content is determined as the target data unit to be processed, and the first data content is determined as the matching data unit;

[0095] S27: Determine if there is a target data unit to be processed that matches the corresponding matching data unit; if yes, proceed to S28; if no, proceed to S29.

[0096] S28: The position of the first byte in the matching data unit in the data to be compressed is taken as the matching position, the length of the matching data unit is taken as the matching length, the target data unit to be processed in the data to be compressed is replaced with the matching position and the matching length, the hash entry corresponding to the hash value of the target data unit to be processed in the hash table is updated based on the position of the first byte in the target data unit to be processed in the data to be compressed, the current processing position is incremented by the matching length, and S23 is re-entered;

[0097] S29: Based on the position of the first byte in the data to be compressed in the data to be processed, update the hash entries corresponding to the hash values ​​of the data to be processed in the hash table, increment the current processing position by the first preset number, and re-enter S23.

[0098] In this embodiment, if no target data unit to be processed matches a corresponding matching data unit, the hash table is updated based on the position of the first byte of each of the multiple target data units in the data to be compressed, and the current processing position is incremented by a first preset number, then the process re-enters S23. If a target data unit to be processed matches a corresponding matching data unit, the matching position and matching length are determined based on the matching data unit, the hash table is updated based on the position of the first byte of the target data unit in the data to be compressed, and the target data unit to be processed in the data to be compressed is replaced with the matching position and matching length. Then, the current processing position is incremented by the matching length, and the process re-enters S23. It should be noted that if multiple target data units to be processed exist, to improve efficiency, the current processing position can be incremented by the matching length corresponding to a target data unit that is very far in the data to be processed.

[0099] The data compression method provided in this application utilizes multiple computing modules to perform parallel matching of duplicate data on multiple data units to be processed, thereby significantly improving the compression performance of the LZ77 algorithm.

[0100] Based on the above embodiments, as a preferred embodiment, the hash table includes multiple hash entries, each hash entry is indexed by a first-level hash value calculated based on a first hash algorithm, each hash entry includes a second preset number of second-level hash values ​​and a second preset number of positions in the data to be compressed, the second-level hash value is the hash value of the data content calculated based on a second hash algorithm, and both the first-level hash value and the second-level hash value are one byte.

[0101] In this embodiment, the result of the hash algorithm in the existing LZ77 hash table technology is reduced from 2 bytes to 1 byte, thus requiring only 2 bytes. 8 = 256 hash table entries, which is the number of existing LZ77 hash table entries (2 16=32768) reduced by 256 times. One problem brought about by the significant reduction in hash entries is the increase in hash collision rate and the decrease in the data matching success rate of LZ77, thus the data compression rate will also decrease. In order to solve the hash collision problem, this embodiment first expands the size of the hash bucket. The existing hash bucket size in the LZ77 hash table is 2, that is, one hash entry can store 2 positions. This embodiment expands the hash bucket size to a second preset number, such as 4, that is, one hash entry can store 4 positions. In addition, this embodiment uses the first-level hash value calculated based on the first hash algorithm as the index of the hash entry, and introduces the second-level hash value calculated based on the second hash algorithm into the hash table to avoid the false matching problem caused by hash collision. That is, each hash entry not only stores the second preset number of positions, but also stores the second preset number of second-level hash values.

[0102] For example, if the second preset value is 4, the structure diagram of the hash table is as follows: Figure 6 As shown, each position is still represented by 2 bytes, and each secondary hash value is represented by 1 byte. Therefore, each hash entry occupies a total of 4*(2+1) = 12 bytes, and the size of the entire hash table is 256*12 = 3072 bytes (3KB). The secondary hash value of the data corresponding to the position stored in pos1 is stored in hash1, the secondary hash value of the data corresponding to the position stored in pos2 is stored in hash2, the secondary hash value of the data corresponding to the position stored in pos3 is stored in hash3, and the secondary hash value of the data corresponding to the position stored in pos4 is stored in hash4.

[0103] Furthermore, based on the hash table provided in this embodiment, see [link to relevant documentation]. Figure 7 , Figure 7 for Figure 5 A detailed flowchart of step S26 is shown below. Figure 7 As shown, step S26 specifically includes:

[0104] S261: Each computing module calculates the target first-level hash value of the corresponding data unit to be processed based on the first hash algorithm, and uses the target first-level hash value as an index to determine the corresponding target hash entry in the hash table;

[0105] S262: Calculate the target secondary hash value of the corresponding data unit to be processed using each computing module based on the second hash algorithm, and determine the candidate matching position corresponding to the target secondary hash value in the target hash entry.

[0106] S263: Using each computing module, read the first data content starting from the candidate matching position in the data to be compressed, and read the second data content starting from the data unit to be processed in the data to be compressed, and match the first data content and the second data content; if the first data content and the second data content match successfully, then determine the data unit to be processed contained in the second data content as the target data unit to be processed, and determine the first data content as the matching data unit.

[0107] In this embodiment, each calculation module performs a duplicate data matching operation on the input data unit to be processed. The specific process is as follows: calculate the target first-level hash value of the data unit to be processed based on the first hash algorithm, determine the corresponding target hash entry in the hash table using the target first-level hash value as an index, calculate the target second-level hash value of the data unit to be processed based on the second hash algorithm, and determine whether the target hash entry stores a valid value corresponding to the target second-level hash value.

[0108] If it does not exist, the match fails. The hash entry corresponding to the hash value of the data unit to be processed in the hash table is updated based on the position of the first byte in the data to be compressed, that is, the target hash entry is updated. Specifically, the target secondary hash value and the position of the first byte in the data unit to be processed in the data to be compressed are stored in the target hash entry.

[0109] If a match exists, the valid value corresponding to the target secondary hash value is read from the target hash entry as a candidate matching position. The first data content is read starting from the candidate matching position in the data to be compressed, and the second data content is read starting from the data unit to be processed in the data to be compressed. Specifically, the second data content is read starting from the first byte of the data unit to be processed in the data to be compressed. The first and second data contents are matched. If the first and second data contents match successfully, the matching position and matching length are determined. Further, the hash entry corresponding to the primary hash value of the target data unit to be processed is updated in the hash table based on the position of the first byte in the target data unit to be processed in the data to be compressed. Specifically, the position of the first byte in the target data unit to be processed in the data to be compressed is stored in the hash entry corresponding to the secondary hash value of the target data unit to be processed.

[0110] Therefore, this implementation reduces the hash table memory overhead of the LZ77 algorithm by greatly reducing the length of the hash table and slightly expanding its width. It can be used in memory-scarce scenarios, such as ASIC or FPGA (Field Programmable Gate Array) hardware implementations of the LZ77 algorithm, and embedded devices with limited memory resources.

[0111] The following describes an application embodiment provided in this application; see [link to application example]. Figure 8 , Figure 8 This is a flowchart illustrating a duplicate data identification technique based on parallel computing and memory optimization according to an exemplary embodiment, such as... Figure 8 As shown, the specific steps include:

[0112] Step 1: Set the first byte of the input data to 0 and set it to the starting address of the current data unit.

[0113] Step 2: Determine if the number of bytes in the current input stream is at least one data unit, that is, greater than or equal to 4 bytes; if yes, proceed to Step 3; otherwise, end the execution and output all matching information.

[0114] Step 3: Starting with each byte in the current data unit, perform four-way parallel hash value calculation. See also... Figure 9 , Figure 9 This is a schematic diagram illustrating a duplicate data identification technique based on parallel computing according to an exemplary embodiment;

[0115] Step 4: Using the four calculated first-level hash values ​​as indices in the hash table, perform a four-way parallel hash table lookup. The lookup process requires comparing the second-level hash values ​​stored in the hash table with the second-level hash values ​​calculated for the current data unit. Only when the second-level hash values ​​are also equal can it be considered a hash match.

[0116] Step 5: If the hash matches, continue comparing the data content, sequentially determining the first truly matched position in paths 1, 2, 3, and 4, obtaining the matching position and matching length, calculating the hash value starting from each matching byte, updating the hash table, and incrementing the starting position of the current data unit by the matching length bytes; if the hash does not match, update the hash table with the current four hash values ​​and position information, and increment the starting position of the current data unit by 4 bytes.

[0117] Step 6: Return to step 2 to proceed to the next round of execution.

[0118] Therefore, this embodiment can significantly improve the performance of data compression algorithms based on LZ77, and can be used to implement high-performance data compression algorithms within storage devices. For a given set of data to be compressed, duplicate data identification can be performed in parallel. Then, for the identified unmatched symbols, matching positions, and matching length sequences, further compression can be performed using techniques such as Huffman coding or FSE (Finite State Entropy) coding.

[0119] The following describes a data compression apparatus provided in an embodiment of this application. The data compression apparatus described below and the data compression method described above can be referred to each other.

[0120] See Figure 10 A structural diagram of a data compression apparatus according to an exemplary embodiment is shown, such as... Figure 10 As shown, it includes:

[0121] Module 10 is used to acquire the data to be compressed;

[0122] The determining module 20 is used to determine the current data unit in the data to be compressed; wherein the current data unit includes a first preset number of bytes;

[0123] Extraction module 30 is used to extract multiple data units to be processed from the data to be compressed, using each byte in the current data unit as the first byte; wherein each data unit to be processed includes a second preset number of bytes;

[0124] Multiple computing modules 40 are used to match multiple data units to be processed with previous data in parallel;

[0125] The replacement module 50 is used to determine the target data unit to be processed that matches the corresponding matching data unit, take the position of the first byte in the matching data unit in the data to be compressed as the matching position, take the length of the matching data unit as the matching length, and replace the target data unit to be processed in the data to be compressed with the matching position and the matching length.

[0126] The data compression device provided in this application embodiment utilizes multiple computing modules to perform parallel matching of duplicate data on multiple data units to be processed, thereby significantly improving the compression performance of the LZ77 algorithm.

[0127] Based on the above embodiments, as a preferred embodiment, the determining module 20 includes:

[0128] The determination submodule is used to determine the current processing position;

[0129] The extraction submodule is used to extract a current data unit containing the first preset number of bytes from the data to be compressed, using the byte corresponding to the current processing position in the data to be compressed as the first byte.

[0130] Correspondingly, it also includes:

[0131] The first increment module is used to increment the current processing position by the first preset number and restart the workflow of the extraction submodule when there is no target data unit to be processed that matches the corresponding matching data unit.

[0132] The second increment module is used to replace the target data unit to be processed in the data to be compressed with the matching position and the matching length, then increment the current processing position by the matching length and restart the workflow of the extraction submodule.

[0133] Based on the above embodiments, as a preferred implementation, the calculation module is specifically used for: calculating the target hash value of the corresponding data unit to be processed; determining the corresponding target hash entry in the hash table using the target hash value as an index; determining the candidate matching position in the target hash entry; reading first data content starting from the candidate matching position in the data to be compressed; reading second data content starting from the data unit to be processed in the data to be compressed; and matching the first data content and the second data content; wherein, the hash table is used to store the correspondence between the hash value of the data content and its position in the data to be compressed;

[0134] Accordingly, the replacement module 50 is specifically used to: when the first data content and the second data content are successfully matched, determine the data unit to be processed contained in the second data content as the target data unit to be processed, and determine the first data content as the matching data unit.

[0135] Based on the above embodiments, as a preferred embodiment, it further includes:

[0136] The first update module is used to update the hash entry corresponding to the hash value of the target data unit to be processed in the hash table based on the position of the first byte in the data to be compressed.

[0137] The second update module is used to update the hash entries corresponding to the hash values ​​of the multiple data units to be processed in the hash table based on the position of the first byte in the data to be compressed when there is no target data unit to be processed that matches the corresponding matching data unit.

[0138] Based on the above embodiments, as a preferred embodiment, it further includes:

[0139] The elimination module is used to eliminate the target data unit that is later in position between the first target data unit and the second target data unit when there is overlap between the second data content containing the first target data unit to be processed and the second data content containing the second target data unit to be processed.

[0140] Based on the above embodiments, as a preferred embodiment, the hash table includes multiple hash entries, each hash entry is indexed by a first-level hash value calculated based on a first hash algorithm, each hash entry includes a second preset number of second-level hash values ​​and a second preset number of positions in the data to be compressed, the second-level hash value is the hash value of the data content calculated based on a second hash algorithm, and both the first-level hash value and the second-level hash value are one byte.

[0141] Based on the above embodiments, as a preferred implementation, the calculation module is specifically used for: calculating the target first-level hash value of the corresponding data unit to be processed based on the first hash algorithm, and determining the corresponding target hash entry in the hash table using the target first-level hash value as an index; calculating the target second-level hash value of the corresponding data unit to be processed based on the second hash algorithm, and determining the candidate matching position corresponding to the target second-level hash value in the target hash entry; reading first data content starting from the candidate matching position in the data to be compressed, reading second data content starting from the data unit to be processed in the data to be compressed, and matching the first data content and the second data content.

[0142] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0143] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 11 This is a structural diagram of an electronic device according to an exemplary embodiment, such as... Figure 11 As shown, the electronic device includes:

[0144] Communication interface 1 enables information exchange with other devices, such as network devices;

[0145] Processor 2 is connected to communication interface 1 to enable information exchange with other devices and, when running a computer program, executes the data compression methods provided by one or more of the above-mentioned technical solutions. The computer program is stored in memory 3.

[0146] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 11 The general will label all buses as Bus System 4.

[0147] The memory 3 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0148] It is understood that memory 3 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 3 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0149] The methods disclosed in the embodiments of this application can be applied to processor 2, or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 2 or by instructions in the form of software. The processor 2 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0150] When processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0151] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 that stores a computer program, which can be executed by a processor 2 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0152] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0153] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data compression method, characterized in that, include: Obtain the data to be compressed; The current data unit is determined in the data to be compressed; wherein the current data unit includes a first preset number of bytes; Using each byte in the current data unit as the first byte, multiple data units to be processed are extracted from the data to be compressed; wherein each data unit to be processed includes a second preset number of bytes; Multiple computing modules are used in parallel to match multiple data units to be processed with previous data; If no target data unit to be processed is found that matches the corresponding matching data unit, the process will re-enter the step of determining the current data unit in the data to be compressed. If no target data unit to be processed is found to match the corresponding matching data unit, then the position of the first byte in the matching data unit in the data to be compressed is taken as the matching position, the length of the matching data unit is taken as the matching length, and the target data unit to be processed in the data to be compressed is replaced with the matching position and the matching length.

2. The data compression method according to claim 1, characterized in that, Determining the current data unit in the data to be compressed includes: Determine the current processing location; Using the byte corresponding to the current processing position in the data to be compressed as the first byte, extract the current data unit containing the first preset number of bytes from the data to be compressed; Accordingly, after using multiple computing modules to match multiple data units to be processed with the previous data in parallel, the process further includes: If there is no target data unit to be processed that matches the corresponding matching data unit, the current processing position is incremented by the first preset number, and the process re-enters the step of extracting the current data unit containing the first preset number of bytes in the data to be compressed, with the byte corresponding to the current processing position in the data to be compressed as the first byte. Accordingly, after replacing the target data unit to be processed in the data to be compressed with the matching position and the matching length, the method further includes: The current processing position is incremented by the matching length, and the process re-enters the step of extracting the current data unit containing the first preset number of bytes from the data to be compressed, using the byte corresponding to the current processing position in the data to be compressed as the first byte.

3. The data compression method according to claim 1, characterized in that, The step of using multiple computing modules to match multiple data units to be processed with previous data in parallel includes: Each computing module calculates the target hash value of the corresponding data unit to be processed. Using the target hash value as an index, the corresponding target hash entry is determined in a hash table. Candidate matching positions are determined within the target hash entries. First data content is read starting from the candidate matching position in the data to be compressed, and second data content is read starting from the data unit to be processed in the data to be compressed. The first data content and the second data content are then matched. The hash table stores the correspondence between the hash values ​​of the data content and their positions in the data to be compressed. Accordingly, determining the target data unit to be processed that matches the corresponding matching data unit includes: If the first data content and the second data content match successfully, the data unit to be processed contained in the second data content is determined as the target data unit to be processed, and the first data content is determined as the matching data unit.

4. The data compression method according to claim 3, characterized in that, After determining the target data unit to be processed that matches the corresponding matching data unit, the process further includes: The hash entry corresponding to the hash value of the target data unit to be processed in the hash table is updated based on the position of the first byte in the data to be compressed. Accordingly, after using multiple computing modules to match multiple data units to be processed with the previous data in parallel, the process further includes: If no target data unit to be processed is found that matches the corresponding matching data unit, then the hash entries corresponding to the hash values ​​of the multiple data units to be processed in the hash table are updated based on the position of the first byte in the data to be compressed.

5. The data compression method according to claim 3, characterized in that, After determining the target data unit to be processed that matches the corresponding matching data unit, the process further includes: If the second data content containing the first target data unit to be processed overlaps with the second data content containing the second target data unit to be processed, then the target data unit that is later in position between the first target data unit to be processed and the second target data unit to be processed will be removed.

6. The data compression method according to claim 3, characterized in that, The hash table includes multiple hash entries. Each hash entry is indexed by a first-level hash value calculated based on a first hash algorithm. Each hash entry includes a second preset number of second-level hash values ​​and a second preset number of positions in the data to be compressed. The second-level hash value is the hash value of the data content calculated based on a second hash algorithm. Both the first-level hash value and the second-level hash value are one byte.

7. The data compression method according to claim 6, characterized in that, The process of calculating the target hash value of the corresponding data unit to be processed using each computing module, determining the corresponding target hash entry in the hash table using the target hash value as an index, and determining the candidate matching position in the target hash entry includes: Each computing module calculates the target first-level hash value of the corresponding data unit to be processed based on the first hash algorithm, and uses the target first-level hash value as an index to determine the corresponding target hash entry in the hash table; Each computing module calculates the target secondary hash value of the corresponding data unit to be processed based on the second hash algorithm, and determines the candidate matching position corresponding to the target secondary hash value in the target hash entry.

8. A data compression device, characterized in that, include: The acquisition module is used to acquire the data to be compressed; A determining module is used to determine the current data unit in the data to be compressed; wherein the current data unit includes a first preset number of bytes; An extraction module is used to extract multiple data units to be processed from the data to be compressed, using each byte in the current data unit as the first byte; wherein each data unit to be processed includes a second preset number of bytes; Multiple computing modules are used to match multiple data units to be processed with previous data in parallel; if no target data unit to be processed is found to match the corresponding data unit, the workflow of the determination module is restarted; if a target data unit to be processed is found to match the corresponding data unit, the workflow of the replacement module is started. The replacement module is used to take the position of the first byte in the matching data unit in the data to be compressed as the matching position, take the length of the matching data unit as the matching length, and replace the target data unit to be processed in the data to be compressed with the matching position and the matching length.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data compression method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data compression method as described in any one of claims 1 to 7.

Images