Data compression method and device, electronic equipment, medium and computer program product

Abstract

The invention discloses a data compression method and device, electronic equipment, a medium and a computer program product, and belongs to the technical field of data processing. The method comprises the steps that data reading is conducted through N threads according to a thread parallel mode, first data are obtained, the data value read by each thread is the data value of K bytes in the first data, and N and K are integers larger than 1; obtaining N forward matching mask sequences, wherein each forward matching mask sequence is used for indicating a matching relationship between the data value read by one thread and the data value read by the previous thread; based on the N forward matching mask sequences and a hash table corresponding to the first data, a first position and a second position are determined, the first position is the initial position of a first data value successfully matched in the forward direction in the first data, and the second position is the initial position of a data value before the first data value and matched with the first data value; and performing data compression based on the first position and the second position.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a data compression method, device, electronic device, medium and computer program product. Background Art

[0002] At present, in order to save resources such as storage space or transmission bandwidth occupied by data, compressing data is a common operation. For example, a compression algorithm can be used to compress data losslessly.

[0003] Exemplarily, taking the lz4 compression algorithm as an example, the electronic device can use the lz4 compression algorithm to sequentially compress each data fragment in a single-threaded manner. Specifically, the electronic device can first read the i-th data fragment, and update the hash table according to the matching result of the hash value of the i-th data fragment in the hash table, and perform a corresponding compression operation on the i-th data fragment; then, the electronic device can read the i+1-th data fragment, and so on, until the compression operation on the last data fragment is completed.

[0004] However, since the related art compresses data in order of the positions of the data fragments, the computing resources of the electronic device are occupied for a long time during the data compression process, resulting in low data compression efficiency of the electronic device. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data compression method, device, electronic device, medium and computer program product, which can improve the data compression efficiency of electronic devices.

[0006] In a first aspect, an embodiment of the present application provides a data compression method, the method comprising: reading data through N threads in a thread parallel manner to obtain first data, the data value read by each thread being a data value of K bytes in the first data, and N and K being integers greater than 1; obtaining N forward matching mask sequences, each forward matching mask sequence being used to indicate a matching relationship between a data value read by a thread and a data value read by a thread sorted earlier; determining a first position and a second position based on the N forward matching mask sequences and a hash table corresponding to the first data, the first position being the starting position of the first data value that successfully forward matches in the first data, and the second position being the starting position of a data value that is before the first data value and matches the first data value; and performing data compression based on the first position and the second position.

[0007] In a second aspect, an embodiment of the present application provides a data compression device, which includes: an acquisition module and a processing module; the acquisition module is used to read data through N threads in a thread parallel manner to obtain first data, the data value read by each thread is the data value of K bytes in the first data, and N and K are both integers greater than 1; the acquisition module is also used to obtain N forward matching mask sequences, each forward matching mask sequence is used to indicate the matching relationship between the data value read by one thread and the data value read by the thread sorted earlier; the processing module is used to determine the first position and the second position based on the N forward matching mask sequences and the hash table corresponding to the first data, the first position being the starting position of the first data value that successfully forward matches in the first data, and the second position being the starting position of the data value that is before the first data value and matches the first data value; the processing module is also used to perform data compression based on the first position and the second position.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, data can be read by N threads in a thread parallel manner to obtain first data, where the data value read by each thread is a data value of K bytes in the first data, and N and K are both integers greater than 1; N forward matching mask sequences are obtained, where each forward matching mask sequence is used to indicate the matching relationship between the data value read by a thread and the data value read by the thread in the previous order; based on the N forward matching mask sequences and the hash table corresponding to the first data, the first position and the second position are determined, where the first position is the starting position of the first data value in the first data that is successfully forward matched, and the second position is the starting position of the data value that is before the first data value and matches the first data value; based on the first position and the second position, data compression is performed. Through this scheme, on the one hand, since N forward matching mask sequences corresponding to N threads can be obtained in a thread parallel manner, the forward matching mask sequence corresponding to each thread can reflect whether the data value read by the thread has a matching forward data value in the first data, that is, the N forward matching mask sequences can characterize whether there is a matching data value inside the first data, thereby improving the efficiency of judging the matching situation between the data values ​​read by the N threads. On the other hand, since the hash table corresponding to the first data can reflect whether there is a matching data value in the hash table for the data value in the first data, the starting matching position (i.e., the first position) in the first data and the matching position (i.e., the second position) corresponding to the starting matching position can be accurately determined according to the N forward matching mask sequences and the hash table corresponding to the first data, thereby improving the accuracy of data compression. In this way, compared with the single-threaded compression method, the data compression method provided in the embodiment of the present application can improve the data compression efficiency through the multi-threaded parallel compression method. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a data compression method provided in an embodiment of the present application;

[0014] Figure 2 It is a schematic diagram of threads involved in the data compression method provided in an embodiment of the present application and data values ​​read by the threads;

[0015] Figure 3 It is a schematic diagram of threads involved in the data compression method provided in an embodiment of the present application and data values ​​read by the threads;

[0016] Figure 4 It is a schematic diagram of threads involved in the data compression method provided in an embodiment of the present application and data values ​​read by the threads;

[0017] Figure 5 It is a flow chart of a data compression method provided in an embodiment of the present application;

[0018] Figure 6 is a structural schematic diagram of a data compression device provided in an embodiment of the present application;

[0019] Figure 7 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0022] The nouns or terms involved in this application specification are explained below.

[0023] 1. Compression sequence: The unit of compressed data. Multiple sequences can constitute compressed data. The compressed data can be used to restore the pre-compression data without loss. The structure of the compression sequence is shown in Table 1:

[0024] Table 1

[0025]

[0026] As shown in Table 1, a Sequence may include two parts: a header (Token) and a payload (Body).

[0027] The header includes a literal length and a matching length, the literal length occupies 4 bits, and the matching length occupies 4 bits.

[0028] Body includes a literal part and a matching part. The literal part can include two parts: "length" and "data". The "length" part is used to fill in the length of the literal data, and the "data" part is used to fill in the literal data.

[0029] The matching part may include two parts: "offset" and "length", wherein the "offset" is used to fill in the position offset of the matching data, and the "length" part is used to fill in the matching length of the matching data.

[0030] Hash calculation: Converting a string of characters into a number through calculation can achieve fast search and save storage space.

[0031] 2. lz4 compression algorithm: a lossless compression algorithm characterized by fast compression and decompression speed. In the lz4 compression algorithm, the hash value of 4-byte literal data and its corresponding position can form a hash table for fast search and matching.

[0032] Specifically, in the lz4 compression algorithm, a single-thread sequential compression method may be used, where one thread sequentially compresses the data values ​​read by each thread in accordance with the order of data positions.

[0033] Compression efficiency: refers to the speed of compressing or decompressing data, that is, the amount of data compressed per unit time.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] The data compression method, device, electronic device, medium and computer program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0036] The data compression method, device, electronic device, medium, and computer program product provided in the embodiments of the present application can be applied to scenarios where lossless or lossy data compression is performed. For example, it can be applied to memory data compression (such as compressing unused memory pages to free up memory), or text, video, or audio file compression, etc., where lossless compression is required.

[0037] The data compression method provided in the embodiment of the present application is executed by a data compression device, which may be an electronic device, or a functional module or entity in an electronic device, and the embodiment of the present application does not limit this. The data compression method provided in the embodiment of the present application will be exemplarily described below using an electronic device as an example.

[0038] The present application embodiment provides a data compression method, Figure 1 A schematic diagram of the process of the data compression method provided in the embodiment of the present application is shown as follows: Figure 1As shown, the data compression method provided in the embodiment of the present application may include the following steps 101 to 104.

[0039] Step 101: An electronic device reads first data through N threads in a thread parallel manner.

[0040] The data value read by each thread may include the data value of K bytes in the first data, and N and K are both integers greater than 1.

[0041] In some embodiments of the present application, K may be 2, 3, 4, 5, 6 or 7, etc.

[0042] For example, in order to improve the compression ratio, K may be an integer greater than 4.

[0043] In some embodiments of the present application, the value of N may be any possible value such as 2, 3, 4, 16, 32, 64, etc.

[0044] In some embodiments of the present application, the N threads may be arranged in ascending order of thread identifiers (tid), that is, the thread identifier of the thread at the front of the order is smaller.

[0045] The thread identifier may also be referred to as a thread number.

[0046] In some embodiments of the present application, the above-mentioned N threads may constitute a thread group. Each thread starts from the starting position of the first data and moves backward by the thread identifier (tid) bytes to read the K-byte data value in the first data. The value range of the tid of the thread in the thread group is [0, N-1]. It can be understood that N threads can effectively read N+3 bytes of data value by reading once.

[0047] For example, assuming K = 4, Figure 2 As shown, the starting position of the first data is loc, and the electronic device can control each of the 30 threads to start with loc+tid of each thread and read a 4-byte data value (value), wherein the data value can be referred to as val.

[0048] Specifically, from Figure 2 It can be seen that the first thread can read the data values ​​from position 0 to position 4, the second thread can read the data values ​​from position 1 to position 5, the third thread can read the data values ​​from position 2 to position 6, and so on, the nth thread reads the data values ​​from position n-1 to position n+3. Figure 2 One position in the data corresponds to a byte value, and the value range of n is [0, N-1].

[0049] It can be understood that the tid of the g-th thread among the N threads is g-1, where g is a positive integer less than N.

[0050] In some embodiments of the present application, the N threads mentioned above may be heterogeneous threads of a framework (Open Computing Language, OpenCL) for writing programs based on heterogeneous platforms.

[0051] In some embodiments of the present application, the above N threads can be run in threads in a graphics processing unit (GPU), a neural network processor (NPU) or other non-central processing unit (CPU) devices, so that the CPU computing power can be released. Of course, in some embodiments, the above N threads can also run in the CPU.

[0052] In some embodiments of the present application, when N threads are running on the GPU, N can be configured to 32 or 64, or even larger.

[0053] In some embodiments of the present application, N threads may read data in parallel, such as reading data simultaneously, thereby shortening the total time consumed in reading the first data.

[0054] In some embodiments of the present application, the first data may be memory data in a memory (such as an unused memory page), or data in a text, video, or audio file. For ease of description, the memory data or file including the first data is collectively referred to as second data.

[0055] Step 102: The electronic device obtains N forward matching mask sequences.

[0056] Each forward matching mask sequence is used to indicate a matching relationship between a data value read by a thread and a data value read by a thread that is ordered previously.

[0057] In some embodiments of the present application, the electronic device may determine the order of the N threads according to the thread numbers of the N threads.

[0058] Exemplarily, the N threads may be sorted in ascending order of thread numbers.

[0059] For example, the thread numbers of the 10 threads are 0 to 9, and the threads arranged before thread 3 include thread 0, thread 1, and thread 2.

[0060] It can be understood that the N threads can read data according to the above order.

[0061] In some embodiments of the present application, the matching relationship between a data value read by one thread and a data value read by another thread may include any of the following: data value matching, data value mismatching.

[0062] It should be noted that the above N forward matching mask sequences correspond one-to-one to the N threads.

[0063] In some embodiments of the present application, the forward matching mask sequence may include N mask values, each mask value corresponds to one thread among the N threads. Alternatively, the forward matching mask sequence corresponding to each thread may include N-1 mask values, each mask value corresponds to one thread among the N threads except the current thread; and each mask value indicates whether the data value read by the current thread is the same as that read by the thread corresponding to the mask value.

[0064] It can be understood that in a forward matching mask sequence corresponding to a thread, the mask value corresponding to the thread following the thread indicates that the data value read by the thread following the thread does not match the data value read by the thread.

[0065] It should be noted that data can be shared among the N threads, such as sharing their corresponding forward matching mask sequences.

[0066] In some embodiments of the present application, the above step 102 can be implemented by the following step 102A.

[0067] Step 102A, the electronic device marks, in a thread parallel manner, threads that are sorted before the i-th thread and have the same data value as that read by the i-th thread as a first mask value, and marks threads that are sorted before the i-th thread and have different data values ​​read by the i-th thread as a second mask value, so as to obtain a forward matching mask sequence corresponding to the i-th thread.

[0068] The value range of i can be [2, N].

[0069] It can be understood that for the first thread among the N threads, i.e., the thread with the smallest thread number, since there is no thread arranged before the first thread, it can be directly determined that the corresponding forward matching mask values ​​are all 0. In other words, the electronic device can directly determine the forward matching mask sequence corresponding to the first thread without performing data value matching comparison through threads, which can save computing power.

[0070] For each thread ranked after the first thread among the N threads, the electronic device can perform data matching on the data value read by one thread with the data value read by each thread ranked before the thread in a thread parallel manner to obtain a forward matching mask sequence corresponding to the thread.

[0071] It can be understood that, for the i-th thread among the N threads, the threads ordered before the i-th thread may include: the 1st thread to the (i-1)th thread, where i is a positive integer.

[0072] For example, the threads arranged before the fourth thread include: the first thread, the second thread, and the third thread.

[0073] In some embodiments of the present application, the electronic device can obtain forward matching mask sequences corresponding to N-1 threads with tid values ​​greater than 0 through at least two threads among N threads in a thread parallel manner to obtain N-1 forward matching mask sequences.

[0074] The at least two threads may be any of the following: at least two threads with the smallest thread number, at least two randomly selected threads, at least two threads that first complete data value reading, and at least two preset threads.

[0075] In some embodiments of the present application, each thread of the at least two threads may obtain a forward matching mask sequence corresponding to at least one thread of the N-1 threads.

[0076] In some embodiments of the present application, the electronic device may obtain the corresponding forward matching mask sequences respectively through N-1 threads in a thread parallel manner.

[0077] Specifically, the electronic device can, through the i-th thread, mark the threads that are sorted before the i-th thread and have the same data value as that read by the i-th thread as the first mask value, and mark the threads that are sorted before the i-th thread and have different data values ​​read by the i-th thread as the second mask value, so as to obtain a forward matching mask sequence corresponding to the i-th thread.

[0078] It can be understood that “marking the thread whose data value is the same as that read by the i-th thread among the threads sorted before the i-th thread as the first mask value” may include: marking the thread whose data value is the same as that read by the i-th thread among the threads sorted before the i-th thread as the first mask value. Similarly, “marking the thread whose data value is different from that read by the i-th thread among the threads sorted before the i-th thread as the second mask value” may include: marking the thread whose data value is different from that read by the i-th thread among the threads sorted before the i-th thread as the second mask value.

[0079] For example, assuming that the data value read by the 23rd thread is "abcd", the data value read by the 1st thread is "dfer", and the data value read by the 10th thread is "abcd", the electronic device can record the 1st thread as the second mask value and the 10th thread as the second mask value in the forward matching mask sequence corresponding to the 23rd thread in a thread parallel manner. It should be noted that this example is illustrated by taking the matching result of the data value read by the 23rd thread with the data values ​​read by the 10th and 17th threads as an example. In actual implementation, the electronic device can match the data value read by the 23rd thread with the data values ​​read by the 1st to 22nd threads respectively, and mark a thread with a mask value for each matching data value read by a thread.

[0080] In some embodiments of the present application, the data values ​​read by two threads are the same may include: each byte read by the threads is the same, in other words, if one or more bytes in the data values ​​read by the two threads are different, it means that the data values ​​read by the two threads are different. For example, if the data value read by one thread is "abce" and the data value read by another thread is "abcd", then the data values ​​read by the two threads are different.

[0081] In some embodiments of the present application, a thread may match the data value read by the thread with the data value read by each thread sorted before the thread, or may first convert the data values ​​read by the two threads into hash values, and then compare the two hash values ​​to see if they match. The specific details may be determined based on actual usage requirements.

[0082] In some embodiments of the present application, the first mask value may be 1, and the second mask value may be 0.

[0083] For example, if the data value read by the fifth thread does not match the data values ​​read by all threads before the fifth thread, the forward matching mask sequence corresponding to the fifth thread includes only 0s.

[0084] It can be understood that the i-th thread can match the data value read by the i-th thread with the data value read by the thread whose tid is less than the i-th thread, and the i-th thread can directly mark the thread whose tid is greater than or equal to the i-th thread as the second mask value without performing data matching. This can improve the efficiency of obtaining the forward matching mask sequence corresponding to the thread.

[0085] In this way, since the electronic device can match data values ​​in a thread-parallel manner and mark the threads sorted before the i-th thread to obtain the forward matching mask sequences corresponding to the N-1 threads respectively, the efficiency of obtaining the forward matching mask sequence can be improved. The forward matching mask sequence can characterize whether the data value read by the thread has a matching data value in the first data, thereby improving the efficiency of data compression by the electronic device.

[0086] Step 103: The electronic device determines the first position and the second position based on the N forward matching mask sequences and the hash table corresponding to the first data.

[0087] The first position is the starting position of the first data value that is successfully forward matched in the first data, and the second position is the starting position of the data value that precedes the first data value and matches the first data value.

[0088] In some embodiments of the present application, the hash table corresponding to the first data may include: hash values ​​of data values ​​in the second data that are located before the first data, and position information of these data values ​​in the second data.

[0089] It should be noted that each hash value in the hash table is a hash value converted from K consecutive bytes of data values ​​in the second data, that is, each hash value in the hash table indicates a K-byte data value in the second data, and the position information corresponding to each hash value is the position information of the K-byte data value indicated by the hash value in the second data.

[0090] In some embodiments of the present application, the above hash table is designed so that one hash value corresponds to only one position information, and the position of the last K bytes of data value corresponding to a hash value in the second data is used as the position information filled in the hash table. For example, assuming that the second data includes: efabcdfabcfabcd..., then for "abcd", the position information of the second abcd is filled in the hash table.

[0091] In some embodiments of the present application, the second position may be: a data value read by a thread located before the first thread in the first data, or may be a data value indicated by a hash value in a hash table, which is specifically determined according to a hash table matching result such as a matching result of a data value read by at least one thread in the hash table as described below.

[0092] In some embodiments of the present application, the above step 103 can be implemented by the following step 103A and step 103B.

[0093] Step 103A: The electronic device determines at least one thread from the N threads based on the mask values ​​included in the N forward matching mask sequences.

[0094] In some embodiments of the present application, the electronic device may determine at least one thread from N threads by determining whether N forward matching mask sequences include a first mask value.

[0095] In some embodiments of the present application, if a forward matching mask sequence corresponding to a thread includes a first mask value, it means that a data value read by the thread matches a data value read by the thread corresponding to the first mask value.

[0096] If the forward matching mask sequence corresponding to a thread includes multiple first mask values, it means that the data values ​​read by multiple threads arranged before the thread in the first data match the data value read by the thread.

[0097] In some embodiments of the present application, the electronic device may determine at least one thread from the N threads through at least one thread among the N threads based on the mask values ​​included in the N forward matching mask sequences.

[0098] For example, the electronic device may determine at least one thread from N threads through the first thread based on mask values ​​included in N forward matching mask sequences.

[0099] For example, the electronic device may determine, through the Nth thread, whether each thread belongs to a thread in at least one thread based on a mask value included in a forward matching mask sequence corresponding to each thread.

[0100] In some embodiments of the present application, the above step 103A can be implemented by the following step 103A1 or step 103A2.

[0101] Step 103A1: If M forward matching mask sequences among the N forward matching mask sequences include a first mask value, the electronic device determines the threads corresponding to the M forward matching mask sequences as at least one thread.

[0102] Wherein, M can be a positive integer less than N.

[0103] In one embodiment of the present application, "M forward matching mask sequences among N forward matching mask sequences include the first mask value" means: each of the M forward matching mask sequences corresponds to a thread, and there is a forward matching thread in the first data. Specifically, the data value read by each thread among the M threads matches the data value read by at least one thread arranged before the thread.

[0104] Step 103A2: If none of the N forward matching mask sequences includes the first mask value, the electronic device determines the last thread of the N threads as at least one thread.

[0105] In some embodiments of the present application, "the first mask value is not included in any of the N forward matching mask sequences" can be understood as: each of the N forward matching mask sequences does not include the first mask value, in other words, each of the N forward matching mask sequences contains only the second mask value. In other words, the first data does not include a matching data value, that is, the data values ​​read by the N threads do not match.

[0106] In this way, since the forward matching mask sequence corresponding to a thread includes the first mask value, it can be indicated that: there is a data value in the first data that matches the data value read by the thread, that is, the electronic device can filter at least one thread according to the matching result of the data values ​​read by N threads within the first data, thereby ensuring the filtering accuracy of the at least one thread, so that the first position and the second position can be accurately determined, and the first data can be accurately compressed.

[0107] Step 103B: The electronic device determines the first position and the second position based on the hash table corresponding to the first thread number and the first data.

[0108] The first thread number may be the thread number of the thread with the smallest thread number among the at least one thread determined.

[0109] For example, assuming that the tid of the at least one thread is 5, 15, and 23 respectively, the electronic device determines the first position and the second position based on tid=5 and the hash table.

[0110] It can be understood that when at least one thread is determined through the above step 103A1, the data value read by the thread with the smallest thread number in the at least one thread is: the data value read by the first thread that successfully forward matches within the first data.

[0111] It can be understood that the first position can be referred to as the starting position of the successfully matched data value, and the second position can be the starting position of the data value that matches the successfully matched data value.

[0112] In some embodiments of the present application, the above step 103B may be implemented through the following steps 103B1, 103B2 and 103B3, or may be implemented through the following steps 103B1, 103B4 and 103B5.

[0113] Step 103B1: The electronic device matches data values ​​read by Q threads whose thread numbers are smaller than the first thread number among the N threads with hash values ​​in the hash table.

[0114] Wherein, Q can be a positive integer.

[0115] In some embodiments of the present application, the electronic device can match the data values ​​read by each of the Q threads with the hash values ​​in the hash table through the Q threads in a thread parallel manner. If the data values ​​read by the Q threads are checked in the hash table at the same time, the electronic device can simultaneously check whether there are matching data values ​​in the hash table for the data values ​​read by the Q threads, thereby improving the efficiency of hash matching.

[0116] Specifically, the electronic device can match the data value read by the rth thread with the hash value in the hash table through the rth thread among the Q threads, and can output a matching result. The matching result can indicate whether the data value read by the rth thread finds a matching data value through the hash table, and the value range of r is [1, Q].

[0117] It should be noted that if the hash value of the data value read by a thread has valid location information corresponding to it in the hash table, it means that the data value read by the thread successfully matches the hash value in the hash table. If the hash value of the data value read by a thread does not have valid location information in the hash table, or the hash value of the data value read by the thread does not exist in the hash table, it means that the data value read by the thread fails to match the hash value in the hash table.

[0118] The valid location information may be: the location indicated by the location information is within the location range in the second data.

[0119] For example, assuming that the data value in the second data starts from position 0, and the hash value of the data value read by a thread corresponds to position information "-1" in the hash table, it means that the position information "-1" is outside the position range in the second data, so it can be determined that the data value read by the thread fails to match in the hash table.

[0120] In some embodiments of the present application, the Q threads may first convert the data values ​​they read into hash values, and then use the hash values ​​to match the hash values ​​in the hash table, which can improve the matching efficiency.

[0121] It can be understood that if at least one of the Q threads finds a matching hash value in the hash table, the electronic device can perform the following steps 103B2 and 103B3. If none of the Q threads find a matching hash value in the hash table, then: if M of the N forward matching mask sequences include the first mask value, that is, there is a matching data value inside the first data, the electronic device can perform steps 103B4 and 103B5. If none of the N forward matching mask sequences include the first mask value, that is, there is no matching data value inside the first data, the electronic device can treat the first data as literal data, and perform compression sequence filling, and control the N threads to move backward N positions, and re-execute step 101.

[0122] Step 103B2: If the data values ​​read by L threads among the Q threads match the hash value in the hash table, the electronic device determines the starting position of the data value read by the thread with the smallest thread number among the L threads as the first position.

[0123] Wherein, L is a positive integer less than or equal to Q.

[0124] It can be understood that the above L threads are threads that find matching hash values ​​in the hash table.

[0125] Step 103B3: The electronic device determines the starting position of the data value corresponding to the first hash value in the hash table as the second position.

[0126] The first hash value is a hash value that matches the data value read by the thread with the smallest thread number among the L threads.

[0127] In some embodiments of the present application, assuming that the starting position information of each K-byte data value is associated and stored in the hash table, the electronic device can determine the position indicated by the position information corresponding to the first hash value as the second position, where K is an integer greater than 1.

[0128] Step 103B4: If the data values ​​read by the Q threads do not match the hash values ​​in the hash table, the starting position of the data value read by the thread with the smallest thread number among the at least one thread is determined as the first position.

[0129] In some embodiments of the present application, "data values ​​read by Q threads do not match the hash value in the hash table" means: data values ​​read by all threads before the thread with the smallest thread number in the at least one thread (the thread with a forward matching data value in the first data) do not find matching data values ​​in the hash table, so that the electronic device can determine that the thread with the smallest thread number in the at least one thread is the first thread to successfully match in the first data.

[0130] Simply put, "the data values ​​read by the Q threads do not match the hash values ​​in the hash table" means that in the second data, no data value matching the data value read by the Q threads is found in the data before the first data, that is, the first data appears for the first time in the second data.

[0131] Step 103B5: determine the starting position of the first data value in the first data as the second position.

[0132] The first data value matches the data value read by the thread with the smallest thread number among the at least one thread.

[0133] In some embodiments of the present application, the above embodiments are illustrated by taking the example that the electronic device directly determines the first position and the second position according to the first thread number and the hash table.

[0134] In actual implementation, the electronic device can first determine the starting position of the data value read by the thread with the smallest thread number among the at least one thread (hereinafter referred to as the first thread) as the first position; and determine the starting position of the data value read by the thread that is sorted before the first thread and whose thread number has the smallest difference with the first thread number as the second position.

[0135] For example, Figure 3 As shown, assuming that the thread with the smallest thread number in at least one thread is thread b among N threads, and the mask value corresponding to thread a in the forward matching mask sequence corresponding to thread b is 1, the electronic device can determine the starting position of the data value read by thread b as the first position; and determine the starting position of the data value read by thread a as the second position.

[0136] Furthermore, if the data values ​​read by L threads among the Q threads match the first hash value in the hash table, the first position is updated to the starting position of the data value read by the thread with the smallest thread number among the L threads, and the second position is updated to the starting position of the data value corresponding to the first hash value; otherwise, the first position and the second position are kept unchanged.

[0137] In this way, since the mask value included in each forward matching mask sequence can indicate whether the data value read by a thread matches the data value read by the thread corresponding to the mask value, the at least one thread screened out by the electronic device from the N threads according to the N forward matching mask sequences can accurately reflect the matching situation of the internal data value of the first data, thereby ensuring that the electronic device can accurately determine the first position and the second position according to the starting position and hash table of the data value read by the thread with the smallest thread number in the at least one thread, thereby improving the accuracy of data compression.

[0138] Step 104: The electronic device performs data compression based on the first position and the second position.

[0139] In some embodiments of the present application, the electronic device may compress the target data based on the first position and the second position.

[0140] The starting position of the target data may be the position next to the compression ending position of the previous data compression, which may also be called the compression starting position. The ending position of the target data is determined by the extended matching result of the first data.

[0141] Exemplarily, for the starting position of the target data. If the last byte of data read by the N threads the previous time is compressible data, that is, the data read by the N threads the previous time has been compressed, that is, the starting position of the first data is the next position of the compression end position of the previous data compression, then the starting position of the target data is the same as the starting position of the first data; if all the data read by the N threads the previous time is uncompressed data or there is incompressible data (i.e. literal data) after the compressible data, then: the starting position of the first data is the next position of the end position of the data read by the N threads the previous time, so that the starting position of the target data is smaller than the starting position of the first data, specifically, the starting position of the target data is the starting position of these incompressible data.

[0142] Exemplarily, for the end position of the target data, if the electronic device successfully extends and matches at least one position (each position corresponds to a data value of one byte) from the first data value, the end position of the target data is the end position of the at least one position, for example; if the first position after the first data value fails to extend and match, the end position of the target data is the end position of the first data value.

[0143] In some embodiments of the present application, the target data may include any of the following:

[0144] The first data, that is, the data before the first data, has been compressed;

[0145] Part of the data in the first data, that is, the data values ​​in the first data located after the first data value, still have incompressible data values;

[0146] Data that is located before the first data and has not been compressed (such as uncompressed data values ​​in the data read by N threads previously) and at least part of the data in the first data;

[0147] A data value of at least one position located after the first data and adjacent to the first data matches a data value of at least one target position located after the second position and adjacent to the second position.

[0148] It should be noted that after the electronic device compresses the data based on the first position and the second position, the compression start position may be updated to facilitate subsequent compression. For example, the compression start position may be updated to the next position of the target data.

[0149] It can be understood that the target data is data in the second data, and the target data includes at least part of the data value in the first data.

[0150] In some embodiments of the present application, the electronic device may perform data compression based on the first position and the second position through at least two threads among N threads in a thread parallel manner.

[0151] Exemplarily, the electronic device may perform data compression on the target data based on the first position and the second position through N threads in a thread-parallel manner.

[0152] In some embodiments of the present application, the electronic device can fill in a compression sequence based on the first position and the second position to achieve data compression. The structure of the compression sequence can refer to the relevant description in the above-mentioned glossary.

[0153] In some embodiments of the present application, the electronic device may use the data value between the starting position and the previous position of the first position in the target data as the literal data, and use the data value between the first position and the end position in the target data as the matching data corresponding to the target data. In addition, the electronic device may determine the position offset between the first position and the second position as the matching offset corresponding to the matching data. Thus, the electronic device may fill in the compression sequence corresponding to the target data according to the above Table 1 based on the determined literal data, matching data, literal length, matching length, and matching offset to achieve compression of the target data.

[0154] In some embodiments of the present application, the electronic device may configure each of the at least two threads to fill in the contents of a portion of bytes in the compression sequence to further improve data compression efficiency.

[0155] For example, the electronic device can configure the first thread of N threads to be responsible for filling in the Token part of the sequence, configure the second thread to be responsible for filling in the length of the literal data in the Body part, configure the lower three threads to be responsible for filling in the first 2 bytes of the literal data, configure the fourth thread to be responsible for filling in bytes 3 and 4 of the literal data, and so on, until a thread is assigned to each fillable position corresponding to the compressed sequence.

[0156] For example, when the amount of data in the "data" part of the Body in Table 1 is large, the electronic device can configure each thread in the thread group to fill in a portion of the data value to increase the data filling speed.

[0157] In some embodiments of the present application, the above step 104 can be implemented by the following steps 104A to 104D.

[0158] Step 104A: The electronic device determines the third position based on the first position and the matching offset position corresponding to the jth thread among the N threads in a thread parallel manner, and determines the fourth position based on the second position and the matching offset position.

[0159] Step 104B: The electronic device matches the data value at the third position with the data value at the fourth position to obtain a matching result corresponding to the j-th thread.

[0160] The value range of j can be [1, N].

[0161] In some embodiments of the present application, the electronic device may determine the third position based on the first position and the matching offset position corresponding to the j-th thread through the j-th thread among the N threads in a thread parallel manner, and determine the fourth position based on the second position and the matching offset position. It is understandable that the electronic device may determine N third positions based on the first position and the matching offset positions corresponding to the N threads through the N threads, and determine N fourth positions based on the second position and the matching offset positions corresponding to the N threads. Each thread corresponds to one third position and one fourth position.

[0162] In some embodiments of the present application, the electronic device may match the data value corresponding to the third position with the data value corresponding to the fourth position through N threads in a thread parallel manner to obtain a matching result corresponding to one thread.

[0163] In some embodiments of the present application, the matching offset position corresponding to the jth thread can be determined according to the thread identifier (i.e., thread number) of the jth thread, the number of threads of the N threads, and the current matching iteration count. It should be noted that the matching offset positions corresponding to the N threads are different when the matching iteration count is different.

[0164] For example, the matching offset position corresponding to the jth thread is: the sum of N*d and tidj, where N is the number of threads, d is the current matching iteration number, d is an integer greater than or equal to 0, tidj is the thread identifier of the jth thread, and tidj is an integer greater than or equal to 0.

[0165] In some embodiments of the present application, the electronic device may determine the third position based on the first position, the second position, and the matching offset position corresponding to the j-th thread by using the first formula; and determine the fourth position based on the first position, the second position, and the matching offset position corresponding to the j-th thread by using the second formula.

[0166] The first formula is: Y2 j =first_match_loc±(N*d+tid j );

[0167] The second formula is: Y1 j =match_loc±(N*d+tid j +K).

[0168] In the above first and second formulas, Y1 j Indicates the third position, Y2 j represents the fourth position, first_match_loc represents the first position, match_loc represents the second position, N represents the number of threads of N threads, N is an integer greater than 1, d represents the number of matching iterations, d is an integer greater than or equal to 0, tid j represents the thread identifier of the jth thread; and the “-” in “±” represents forward movement, and the “+” in “±” represents backward movement; (N*d+tid j ) represents the matching offset position corresponding to the jth thread, and (N*d+tid j ) where “+” represents the sum, K is the number of data values ​​read by the thread, and K is an integer greater than or equal to 4.

[0169] It can be understood that according to the above first formula and second formula, the third position can include the first sub-position and the second sub-position; the fourth position can include the third sub-position corresponding to the first sub-position and the fourth sub-position corresponding to the second sub-position. That is, in the dth matching iteration process, the electronic device can replace the position first_match_loc-(N*d+tid j ) data value and position match_loc-(N*d+tid j ) data value and set the position first_match_loc+(N*d+ti j d+K) data value and position match_loc+(N*d+ti j d+K) to obtain the matching result corresponding to the j-th thread.

[0170] It can be understood that each matching iteration may include a forward iteration and a backward iteration. The electronic device may match N positions forward based on the first position, the second position, and the matching offset positions corresponding to the N threads in a thread parallel manner; and may match N positions backward based on the first position, the second position, and the matching offset positions corresponding to the N threads in a thread parallel manner. The order of the forward iteration and the backward iteration may not be limited, that is, the forward iteration may be performed first and then the backward iteration, or the backward iteration may be performed first and then the forward iteration.

[0171] Exemplarily, taking the example that the electronic device determines the corresponding matching results through N threads respectively, in the first matching iteration process, the electronic device can match the data value of the first position before the first position with the data value of the first position before the second position through the thread with tid=0; and match the data value of the second position before the first position with the data value of the second position before the second position through the thread with tid=1; and by analogy, the electronic device matches the data value of the Nth position before the first position with the data value of the Nth position before the second position through the thread with tid=N-1. That is, through N threads, it can be determined whether the data values ​​of the N positions before the first position have corresponding matching data, and N forward expansion matching results are obtained.

[0172] Then, the electronic device can match the data value of the first position after the first position with the data value of the first position after the second position through the thread with tid=0; and match the data value of the second position after the first position with the data value of the second position after the second position through the thread with tid=1; and so on, the electronic device matches the data value of the Nth position after the first position with the data value of the Nth position after the second position through the thread with tid=N-1. That is, it can be determined whether the data values ​​of the N positions after the first position have corresponding matching data through N threads to obtain N backward expansion matching results.

[0173] It can be seen that in each matching iteration, a thread can obtain a forward extension matching result and a backward extension matching result. It can be understood that the end position of the target data is determined by the backward matching result of the last round of backward matching.

[0174] Step 104C: The electronic device determines a first matching length based on the matching results corresponding to the N threads.

[0175] It can be understood that the first data matching length can be the length of the data that can be compressed in the second data.

[0176] In some embodiments of the present application, the above step 104C can be implemented by the following step 104C1 or step 104C2.

[0177] Step 104C1: If there is a position where the matching fails in the matching results corresponding to the N threads, the electronic device determines the data length between the fifth position and the sixth position as the first matching length.

[0178] The fifth position is the last position before the first position where the match fails, and the sixth position is the first position after the first position where the match fails. Specifically, the first matching length may include: positions starting from the first position after the fifth position and ending at the first position before the sixth position.

[0179] For example, assuming that the fifth position is position a1 and the sixth position is position a2+A, the first matching length is from position a1+1 to position a2+A-1.

[0180] In some embodiments of the present application, the locations where matching failures exist in the matching results corresponding to the N threads may include: the locations where matching failures exist in the matching results of the forward matching corresponding to the N threads, and the locations where matching failures exist in the matching results of the backward matching corresponding to the N threads.

[0181] Step 104C2: If the data values ​​in the matching results corresponding to the N threads are all matched, the electronic device updates the matching offset positions corresponding to the N threads based on the number of matching iterations, and determines the first matching length based on the updated matching offset positions.

[0182] In some embodiments of the present application, the electronic device updates the matching offset positions corresponding to N threads based on the matching iteration number, which may include: the electronic device adds 1 to the matching iteration number, and then re-determines the matching offset positions corresponding to the N threads based on the updated matching iteration number.

[0183] It should be noted that the electronic device determines the first matching length based on the updated matching offset position, which may include: the electronic device updates the third position based on the first position and the updated matching offset position through the jth thread among N threads in a thread parallel manner; and updates the fourth position based on the second position and the updated matching offset position.

[0184] Then, the electronic device matches the updated data value of the third position with the updated data value of the fourth position to obtain the latest matching result corresponding to the jth thread. Then, the electronic device can determine whether there is a position where the match fails in the latest matching results corresponding to the N threads, and determine whether it is time to determine the first matching length according to the determination result.

[0185] Specifically, if there is a position where the matching fails in the matching results obtained by the N threads most recently, the matching offset position corresponding to each thread continues to be updated, otherwise step 104C1 continues to be executed, otherwise step 104C2 continues to be executed until there is a position where the matching fails in the matching results obtained by the N threads.

[0186] In this way, on the one hand, since the electronic device can use the position between the last position where the match fails before the first position and the first position where the match fails after the first position as the first matching length when there is a matching position in the matching results corresponding to N threads, the maximum length of the data that can be compressed can be accurately determined, thereby improving the data compression rate.

[0187] On the other hand, when all the matching results corresponding to N threads are matched, it means that there are more data with forward matching data, so the electronic device updates the matching offset positions corresponding to the N threads according to the number of matching iterations, and the first matching length determined based on the updated matching offset position can be the maximum matching length, thereby improving the data compression rate.

[0188] Step 104D: The electronic device performs data compression based on the first position, the second position and the first matching length.

[0189] It should be noted that after the electronic device determines the first matching length, it can also update the first position and the second position, and perform data compression according to the first matching length and the updated first position and the second position. The electronic device can update the first position to: the position of the last failed match in the last round of forward matching results, such as the fifth position mentioned above; and update the second position to: the position of the first failed match in the last round of backward matching results, such as the sixth position mentioned above.

[0190] In some embodiments of the present application, the electronic device may fill in the compression sequence corresponding to the target data based on the first position, the second position and the first matching length to achieve compression of the target data.

[0191] Specifically, the electronic device can use the data value between the starting position and the previous position of the updated first position in the target data as the literal data, and use the data value between the updated first position and the ending position in the target data as the matching data corresponding to the target data. In addition, the electronic device can determine the position offset between the updated first position and the updated second position as the matching offset corresponding to the matching data. Thus, the electronic device can fill in the compression sequence corresponding to the target data according to the above Table 1 based on the determined literal data, matching data, literal length, matching length, and matching offset to achieve compression of the target data.

[0192] For other descriptions of step 104D, refer to the description of step 104 in the above embodiment and the related description of lz4 in the related art.

[0193] In this way, since the electronic device can perform extended matching on the first data in a thread-parallel manner, and each thread performs extended matching on an extended matching position in the first data, compared with the single-thread extended matching solution, the multi-thread parallel extended matching method provided in the embodiment of the present application can improve the extended matching efficiency, thereby further improving the data compression efficiency.

[0194] In the data compression method provided in the embodiment of the present application, on the one hand, since N forward matching mask sequences corresponding to N threads can be obtained in a thread parallel manner, the forward matching mask sequence corresponding to each thread can reflect whether the data value read by the thread has a matching forward data value in the first data, that is, the N forward matching mask sequences can characterize whether there is a matching data value inside the first data, so the judgment efficiency of the matching situation between the data values ​​read by the N threads can be improved. On the other hand, since the hash table corresponding to the first data can reflect whether the data value in the first data has a matching data value in the hash table, the starting matching position (i.e., the first position) in the first data and the matching position (i.e., the second position) corresponding to the starting matching position can be accurately determined according to the N forward matching mask sequences and the hash table corresponding to the first data, thereby improving the accuracy of data compression. In this way, compared with the single-thread compression method, the data compression method provided in the embodiment of the present application can improve the data compression efficiency through the multi-thread parallel compression method.

[0195] In some embodiments of the present application, after the electronic device obtains the hash values ​​of the data values ​​read by N threads, the electronic device may also update the hash table through the N threads based on the forward matching mask sequences corresponding to the N threads.

[0196] It can be understood that when updating the hash table, each thread only updates the location information corresponding to the hash value of its own read data value, and uses the matching mask idea to confirm whether there are other threads with the same hash value as its own thread. If there is later data, the thread will not update to the hash table, otherwise it is the last one and can be updated to the hash table.

[0197] For example, Figure 4 As shown, assuming that the hash value of the data value read by thread a is the same as the hash value of the data value read by thread b, the electronic device can store the hash value of the data value read by b and the location information of thread b in the hash table.

[0198] In some embodiments of the present application, before the electronic device reads data in the second data through N threads, the electronic device can first initialize the hash table corresponding to the second data through N threads. It can be understood that the second data and the first data correspond to the same hash table, and the first data is a data block in the second data.

[0199] Specifically, the electronic device can initialize the hash table through the fth thread among the N threads, starting from the tid position of the fth thread, and each thread moves backward N positions each time, and the value range of f is [1, N].

[0200] Among them, the key variables of the hash table are initialized: the starting position ip_start of the data to be compressed and the starting position loc of the thread group reading, both moving 1 byte each time.

[0201] The complete process of the data compression method provided in the embodiment of the present application is described below.

[0202] The embodiment of the present application provides a concurrent data compression method based on the lz4 compression algorithm, which can support multi-threaded compression of a piece of data. Assume that there are N threads in the multi-threaded system, each thread is numbered tid, and the value range of tid is 0 to N-1. Figure 5 As shown, the data compression algorithm may specifically include the following steps, and each step may be processed in multiple threads (i.e., threads are parallel), and the next step needs to wait for the previous step to be completed before it can be executed. The following focuses on the implementation ideas of the concurrent compression method:

[0203] Step 501: The electronic device initializes a hash table, a starting position ip_start of data to be compressed, and a data reading starting position loc of the N threads through N threads.

[0204] In some embodiments of the present application, the initialization of the hash table may be implemented in a thread-parallel manner, where each thread is initialized starting from the tid position and moving backward n positions each time.

[0205] The starting position ip_start of the data to be compressed and the starting position loc of the data read by N threads are also initialized by N threads concurrently, and each thread moves backward 1 byte each time.

[0206] It can be understood that the above ip_start corresponds to the compression starting position in the above embodiment, and the above loc corresponds to the starting position of the first data in the above embodiment.

[0207] Step 502: The electronic device uses each of the N threads to read the 4-byte data val starting from the loc position + tid position of the data to be compressed to obtain the first data.

[0208] It should be noted that the “+” in step 502 means moving backward.

[0209] Understandably, see Figure 2 , N threads can effectively read N plus 3 bytes of data at a time.

[0210] Step 503: The electronic device searches for a match in the first data through N threads.

[0211] Specifically, the electronic device obtains the forward matching mask sequence corresponding to each thread through each thread in a thread parallel manner, and updates the matching position and the matching corresponding position corresponding to the previous compression sequence based on the N forward matching mask sequences.

[0212] It can be understood that the updated matching position in step 503 can correspond to the starting position of the data value read by the thread corresponding to the "first thread number" in the above embodiment, and the updated matching corresponding position is the "starting position of the first data value" in the above embodiment.

[0213] It can be understood that the meaning of step 503 is that the electronic device can update the first position and the second position most recently determined by the electronic device through the data values ​​currently read by the N threads.

[0214] In some embodiments of the present application, the process of an electronic device obtaining a forward matching mask sequence corresponding to a thread through a thread may be:

[0215] For each of the N threads, a thread may construct a forward matching mask sequence corresponding to the thread in the manner of one mask bit (the bit may be 0 or 1) for each thread tid except for itself.

[0216] Specifically, if the val value of the corresponding thread (i.e., the thread before the current thread) is the same as the val value of the current thread, the mask value of the corresponding thread is recorded as 1, otherwise the mask value of the corresponding thread is recorded as 0. Since only forward matching is found, no matching is found for threads with tid>=current thread, and they are all defaulted to 0. Thus, each thread can obtain a forward matching mask sequence for the val value of the current thread. It can be understood that the mask values ​​in the forward matching mask sequence corresponding to the first thread among N threads, that is, the thread with the smallest thread number among N threads, can be defaulted to 0, that is, no data value comparison is required.

[0217] In some embodiments of the present application, if there is a non-zero mask value in the forward matching mask sequence obtained by N threads, it is considered that the intra-group match is successful; then, the electronic device can determine the starting position of the thread val with a non-zero matching mask and the smallest tid as the starting position of the successful match, that is, the updated first position (recorded as first_match_loc), and the corresponding matched forward process val position is the matching corresponding position match_loc, that is, the updated second position, see Figure 3 .

[0218] If there is no non-zero mask value in the forward matching mask sequence obtained by the N threads, it is considered that the intra-group matching fails, so the electronic device can update the first position first_match_loc to the starting position of the last thread val among the N threads, and of course, it can also be updated to the ending position of the last thread val.

[0219] Step 504: The electronic device determines the final matching position and the matching corresponding position by matching the data value read by the thread located before the first position with the matching result in the hash table.

[0220] It should be noted that even if the match in step 503 is successful, there may be a situation where the corresponding value of val with a smaller tid has no match in the thread group, but there is a match in the hash table, so even if the previous step is successful, it is necessary to continue to search for a match in the hash table.

[0221] In some embodiments of the present application, the electronic device may perform hash calculations on respective vals through N threads to obtain hash values ​​of the val values ​​of the N threads.

[0222] Then, for data whose val position is earlier than first_match_loc (i.e., the data values ​​read by Q threads in the above embodiment), the electronic device can search in the hash table for each thread before first_match_loc whether there is a hash value corresponding to the valid position information that is consistent with the val corresponding to each thread.

[0223] If there are one or more threads (i.e., L threads in the above embodiment) that successfully match in the hash table, it is determined that the thread val of the one or more threads has a match in the hash table, so that the electronic device can update first_match_loc to the starting position of val of the first successfully matched thread in the thread group in the hash table (i.e., the thread with the smallest thread number in the L threads), and update match_loc to the position indicated by the position information found in the hash table for the first successfully matched thread (i.e., the starting position of the data value corresponding to the first hash value in the above embodiment).

[0224] It should be noted that if the match_loc in the hash table has valid position information, it means that the out-of-group match is successful, otherwise the match fails.

[0225] Further, if the electronic device finally determines match_loc and first_match_loc, the electronic device may continue to perform step 505', otherwise the electronic device may continue to perform step 505".

[0226] Step 505 ′: the electronic device may update the position information of match_loc in the hash table to the position information of first_match_loc, and insert the hash values ​​and corresponding position information of val of other threads in the N threads into the hash table.

[0227] It should be noted that after executing step 505", the electronic device may continue to execute step 508.

[0228] In some embodiments of the present application, after N threads obtain the hash value of their respective vals, the electronic device may update the hash table through the N threads.

[0229] The electronic device can update the hash table through N threads. Each thread only updates the hash corresponding to the val it reads. The hash table is designed so that one hash value corresponds to only one position, and the last position is the final position filled in the hash table.

[0230] Specifically, a thread can use the matching mask idea to confirm whether there are other threads in the N threads with the same hash value as its own thread. If there is later data (i.e. data at a later position, such as whether there is later data in the first data that matches the data value read by each thread can be determined based on the thread numbers of the N threads), the position information of the data value read by the thread will not be updated to the hash table. Otherwise, it can be updated to the hash table because it is the last one. Or if a data value read by a thread does not find a match in the hash table, and there is no matching data value in the first data, the position information of the data value read by the thread can be updated to the hash table.

[0231] Step 505 : The electronic device inserts the hash values ​​and position information of val of the N threads into the hash table.

[0232] In some embodiments of the present application, after the electronic device executes step 505', it continues to execute step 506. After the electronic device executes step 505", the electronic device can control the N threads to move positions, such as moving N positions backward.

[0233] Step 506: The electronic device searches forward and backward through N threads based on match_loc and first_match_loc to determine whether there is a longer match length in the first data.

[0234] Specifically, each thread compares the 1-byte data of match_loc±(N*d+tid) and first_match_loc±(N*d+tid) to see if they are consistent. After each comparison, each thread synchronizes the result to the thread group, that is, N threads share the matching result. d is the number of iterations, starting from 0 and increasing by 1 each time.

[0235] If the forward (or backward) matching results of N threads are all successful, d is accumulated and the extended matching continues forward (or backward) for the next cycle. Otherwise, if the matching result of one of the threads is a match failure, the forward search ends. When there are positions where the match fails in both the forward and backward searches, the electronic device can end the extended search. The final matching position is the position of the last failed match forward and the position of the first failed match backward. Note that the forward match cannot exceed ip_start. The head and tail interval is the match length match_length. In other words, the final match length is: (first_match_loc is the position of the last failed match forward, first_match_loc is the position of the first failed match backward).

[0236] It is understood that after the electronic device determines the final match length, it can update first_match_loc and match_loc again to obtain the latest first_match_loc and the latest match_loc. It is understood that the "latest" here refers to the most recently updated or determined.

[0237] Step 507: The electronic device fills the compressed sequence according to match_loc, first_match_loc and the matching length.

[0238] Among them, literal_length = the length between the latest first_match_loc and ip_start, and the data is the content from ip_start to the latest first_match_loc. The match length is obtained by step 506, and the match offset offset = the length between the latest first_match_loc and the latest match_loc. After obtaining this information, the compressed sequence shown in Table 1 can be filled. That is, the first_match_loc and match_loc obtained by the most recent update are used in step 507.

[0239] In some embodiments of the present application, the electronic device can fill each part of the compressed sequence in a thread parallel manner. For example, taking the literal part of the compressed sequence filled in the thread parallel manner as an example, each thread fills the literal data moved backward by tid positions from ip_start, and each thread moves backward by N bytes each time until reaching the latest match_loc.

[0240] It is understandable that the matching length may be stored in both the token and the body. The storable length of the token part is less than 15. If it is less than 15, the variable length part of the body is not used. Otherwise, every 255 is filled with 0xff to reduce the matching length to less than 255. The length of the variable length part required can be calculated in advance, and each thread moves backward N bits each time to perform parallel filling in a similar way to the literal part (i.e., thread parallelism).

[0241] Step 508: The electronic device controls the N threads to move positions.

[0242] Step 509: The electronic device determines whether the moved position corresponds to the last compression sequence in the data to be compressed.

[0243] It can be understood that the electronic device can determine whether the moved position corresponds to the last compression sequence in the data to be compressed, that is, whether the moved position is the last data block of the data to be compressed, based on whether the position after the N threads move includes a valid data value. For example, if at least part of the position after the N threads move is outside the position range of the data to be compressed, it is determined to correspond to the last compression sequence of the data to be compressed, otherwise it is determined not to correspond to the last compression sequence of the data to be compressed. Of course, in actual implementation, it can also be determined whether the moved position is the last compression sequence in other ways.

[0244] It can be understood that if the position after the N threads move is the last compression sequence in the data to be compressed, the electronic device can use the data value read from the position after the N threads move as literal data, fill in a compression sequence, and end the compression process; otherwise, execute the above step 502 again until the compression of the last compression sequence in the data to be compressed is completed.

[0245] It should be noted that the data compression method provided in the embodiment of the present application can be used for memory compression of a mobile phone, or can be used in a series of scenarios requiring lossless compression, such as file compression.

[0246] In the data compression method provided in the embodiment of the present application, a block of data can be compressed through multi-threading by running on a GPU, NPU or other devices through languages ​​such as OpenCL, thereby reducing the running time, increasing the compression speed, and releasing the CPU computing power. For user experience, the efficiency of memory recycling can be improved, and recycling can be run on non-CPUs, so that the more tense CPU computing power can be given to application execution, thereby improving the smooth experience of the mobile phone.

[0247] The data compression method provided in the embodiment of the present application can be executed by a data compression device. In the embodiment of the present application, the data compression device provided in the embodiment of the present application is described by taking the data compression method executed by the data compression device as an example.

[0248] The present application embodiment provides a data compression device, such as Figure 6 As shown, the data compression device 600 includes: an acquisition module 601 and a processing module 602; the acquisition module 601 is used to read data through N threads in a thread parallel manner to obtain first data, the data value read by each thread is the data value of K bytes in the first data, and N and K are both integers greater than 1; the acquisition module 601 is also used to obtain N forward matching mask sequences, each forward matching mask sequence is used to indicate the matching relationship between the data value read by one thread and the data value read by the thread sorted before; the processing module 602 is used to determine the first position and the second position based on the N forward matching mask sequences and the hash table corresponding to the first data, the first position is the starting position of the first data value that successfully forward matches in the first data, and the second position is the starting position of the data value before the first data value and matching the first data value;

[0249] The processing module 602 is further configured to perform data compression based on the first position and the second position.

[0250] In some embodiments of the present application, the acquisition module 601 is specifically used to mark, in a thread parallel manner, threads that are sorted before the i-th thread and have the same data value as that read by the i-th thread as a first mask value, and mark threads that are sorted before the i-th thread and have different data values ​​read by the i-th thread as a second mask value, so as to obtain a forward matching mask sequence corresponding to the i-th thread;

[0251] The value range of i is [2, N].

[0252] In some embodiments of the present application, the processing module 602, the substrate is used for:

[0253] Determine at least one thread from the N threads based on the mask values ​​included in the N forward matching mask sequences;

[0254] The first position and the second position are determined based on a first thread number and the hash table, wherein the first thread number is a thread number of a thread with a smallest thread number among the at least one thread.

[0255] In some embodiments of the present application, the processing module 602 is specifically used to:

[0256] If M forward matching mask sequences among the N forward matching mask sequences include the first mask value, threads corresponding to the M forward matching mask sequences are determined as the at least one thread, where M is a positive integer less than N;

[0257] If none of the N forward matching mask sequences includes the first mask value, the last thread of the N threads is determined as the at least one thread.

[0258] In some embodiments of the present application, the processing module 602 is specifically used to:

[0259] Matching the data values ​​read by Q threads whose thread numbers are smaller than the first thread number among the N threads with the hash values ​​in the hash table, where Q is a positive integer;

[0260] If the data values ​​read by L threads among the Q threads match the hash value in the hash table, the starting position of the data value read by the thread with the smallest thread number among the L threads is determined as the first position, where L is a positive integer less than or equal to Q;

[0261] The starting position of the data value corresponding to the first hash value in the hash table is determined as the second position, wherein the first hash value is a hash value matching the data value read by the thread with the smallest thread number among the L threads.

[0262] In some embodiments of the present application, the processing module 602 is specifically used to:

[0263] In a thread parallel manner, a third position is determined based on the first position and a matching offset position corresponding to a j-th thread among the N threads, and a fourth position is determined based on the second position and the matching offset position, where the value range of j is [1, N];

[0264] Matching the data value at the third position with the data value at the fourth position to obtain a matching result corresponding to the j-th thread;

[0265] Determine a first matching length based on the matching results corresponding to the N threads;

[0266] Data compression is performed based on the first position, the second position and the first matching length.

[0267] In some embodiments of the present application, the processing module 602 is specifically used to:

[0268] If there is a position where the match fails in the matching results corresponding to the N threads, the data length between the fifth position and the sixth position is determined as the first matching length, the fifth position is the last position where the match fails before the first position, and the sixth position is the first position where the match fails after the first position;

[0269] or,

[0270] If the data values ​​in the matching results corresponding to the N threads are all matched, the matching offset positions corresponding to the N threads are updated based on the number of matching iterations, and the first matching length is determined based on the updated matching offset positions.

[0271] In the data compression device provided in the embodiment of the present application, on the one hand, since N forward matching mask sequences corresponding to N threads can be obtained in a thread parallel manner, the forward matching mask sequence corresponding to each thread can reflect whether the data value read by the thread has a matching forward data value in the first data, that is, the N forward matching mask sequences can characterize whether there is a matching data value inside the first data, so the judgment efficiency of the matching situation between the data values ​​read by the N threads can be improved. On the other hand, since the hash table corresponding to the first data can reflect whether the data value in the first data has a matching data value in the hash table, the starting matching position (i.e., the first position) in the first data and the matching position (i.e., the second position) corresponding to the starting matching position can be accurately determined according to the N forward matching mask sequences and the hash table corresponding to the first data, thereby improving the accuracy of data compression. In this way, compared with the single-thread compression method, the data compression method provided in the embodiment of the present application can improve the data compression efficiency through the multi-thread parallel compression method.

[0272] The data compression device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0273] The data compression device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0274] The data compression device provided in the embodiment of the present application can achieve Figures 1 to 5 The various processes implemented by the method embodiment achieve the same technical effect and will not be described again here to avoid repetition.

[0275] Alternatively, if Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned data compression method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0276] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0277] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0278] The electronic device 1500 includes but is not limited to: a radio frequency unit 1501, a network module 1502, an audio output unit 1503, an input unit 1504, a sensor 1505, a display unit 1506, a user input unit 1507, an interface unit 1508, a memory 1509, and a processor 1510 and other components.

[0279] Those skilled in the art will appreciate that the electronic device 1500 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0280] The processor 1510 is configured to read data through N threads in a thread parallel manner to obtain first data, wherein the data value read by each thread is a data value of K bytes in the first data, and N and K are both integers greater than 1;

[0281] The processor 1510 is further configured to obtain N forward matching mask sequences, each of which is used to indicate a matching relationship between a data value read by a thread and a data value read by a thread that is previously sorted;

[0282] The processor 1510 is configured to determine a first position and a second position based on the N forward matching mask sequences and a hash table corresponding to the first data, wherein the first position is a starting position of a first data value in the first data that is successfully forward matched, and the second position is a starting position of a data value that is before the first data value and matches the first data value;

[0283] The processor 1510 is further configured to perform data compression based on the first position and the second position.

[0284] In some embodiments of the present application, the processor 1510 is specifically configured to mark, in a thread parallel manner, a thread that is sorted before the i-th thread and has the same data value as that read by the i-th thread as a first mask value, and mark a thread that is sorted before the i-th thread and has a different data value than that read by the i-th thread as a second mask value, so as to obtain a forward matching mask sequence corresponding to the i-th thread;

[0285] The value range of i is [2, N].

[0286] In some embodiments of the present application, the processor 1510, the substrate is used for:

[0287] Determine at least one thread from the N threads based on the mask values ​​included in the N forward matching mask sequences;

[0288] The first position and the second position are determined based on a first thread number and the hash table, wherein the first thread number is a thread number of a thread with a smallest thread number among the at least one thread.

[0289] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0290] If M forward matching mask sequences among the N forward matching mask sequences include the first mask value, threads corresponding to the M forward matching mask sequences are determined as the at least one thread, where M is a positive integer less than N;

[0291] If none of the N forward matching mask sequences includes the first mask value, the last thread of the N threads is determined as the at least one thread.

[0292] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0293] Matching the data values ​​read by Q threads whose thread numbers are smaller than the first thread number among the N threads with the hash values ​​in the hash table, where Q is a positive integer;

[0294] If the data values ​​read by L threads among the Q threads match the hash value in the hash table, the starting position of the data value read by the thread with the smallest thread number among the L threads is determined as the first position, where L is a positive integer less than or equal to Q;

[0295] The starting position of the data value corresponding to the first hash value in the hash table is determined as the second position, wherein the first hash value is a hash value matching the data value read by the thread with the smallest thread number among the L threads.

[0296] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0297] In a thread parallel manner, a third position is determined based on the first position and a matching offset position corresponding to a j-th thread among the N threads, and a fourth position is determined based on the second position and the matching offset position, where the value range of j is [1, N];

[0298] Matching the data value at the third position with the data value at the fourth position to obtain a matching result corresponding to the j-th thread;

[0299] Determine a first matching length based on the matching results corresponding to the N threads;

[0300] Data compression is performed based on the first position, the second position and the first matching length.

[0301] In some embodiments of the present application, the processor 1510 is specifically configured to:

[0302] If there is a position where the match fails in the matching results corresponding to the N threads, the data length between the fifth position and the sixth position is determined as the first matching length, the fifth position is the last position where the match fails before the first position, and the sixth position is the first position where the match fails after the first position;

[0303] or,

[0304] If the data values ​​in the matching results corresponding to the N threads are all matched, the matching offset positions corresponding to the N threads are updated based on the number of matching iterations, and the first matching length is determined based on the updated matching offset positions.

[0305] In the data compression device provided in the embodiment of the present application, on the one hand, since N forward matching mask sequences corresponding to N threads can be obtained in a thread parallel manner, the forward matching mask sequence corresponding to each thread can reflect whether the data value read by the thread has a matching forward data value in the first data, that is, the N forward matching mask sequences can characterize whether there is a matching data value inside the first data, so the judgment efficiency of the matching situation between the data values ​​read by the N threads can be improved. On the other hand, since the hash table corresponding to the first data can reflect whether the data value in the first data has a matching data value in the hash table, the starting matching position (i.e., the first position) in the first data and the matching position (i.e., the second position) corresponding to the starting matching position can be accurately determined according to the N forward matching mask sequences and the hash table corresponding to the first data, thereby improving the accuracy of data compression. In this way, compared with the single-thread compression method, the data compression method provided in the embodiment of the present application can improve the data compression efficiency through the multi-thread parallel compression method.

[0306] It should be understood that in the embodiment of the present application, the input unit 1504 may include a graphics processor (Graphics Processing Unit, GPU) 15041 and a microphone 15042, and the graphics processor 15041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1506 may include a display panel 15061, and the display panel 15061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include two parts: a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0307] The memory 1509 can be used to store software programs and various data. The memory 1509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1509 may include a volatile memory or a non-volatile memory, or the memory 1509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0308] The processor 1510 may include one or more processing units; optionally, the processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1510.

[0309] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned data compression method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0310] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0311] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data compression method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0312] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0313] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned data compression method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0314] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0315] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0316] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data compression method, characterized in that: The method comprises: In a thread parallel manner, data is read by N threads to obtain first data, where the data value read by each thread is a data value of K bytes in the first data, and N and K are both integers greater than 1; Obtain N forward matching mask sequences, each forward matching mask sequence is used to indicate a matching relationship between a data value read by a thread and a data value read by a thread that is sorted previously; Determine a first position and a second position based on the N forward matching mask sequences and the hash table corresponding to the first data, wherein the first position is the starting position of the first data value in the first data that is successfully forward matched, and the second position is the starting position of the data value that is before the first data value and matches the first data value; Data compression is performed based on the first position and the second position.

2. The method according to claim 1, characterized in that The obtaining of N forward matching mask sequences comprises: In a thread parallel manner, a thread that is sorted before the i-th thread and has the same data value as that read by the i-th thread is marked as a first mask value, and a thread that is sorted before the i-th thread and has a different data value than that read by the i-th thread is marked as a second mask value, so as to obtain a forward matching mask sequence corresponding to the i-th thread; The value range of i is [2, N].

3. The method according to claim 1 or 2, characterized in that: The determining the first position and the second position based on the N forward matching mask sequences and the hash table corresponding to the first data includes: Determine at least one thread from the N threads based on the mask values ​​included in the N forward matching mask sequences; The first position and the second position are determined based on a first thread number and the hash table, wherein the first thread number is a thread number of a thread with a smallest thread number among the at least one thread.

4. The method according to claim 3, characterized in that The determining at least one thread from the N threads based on the mask values ​​included in the N forward matching mask sequences comprises: If M forward matching mask sequences among the N forward matching mask sequences include the first mask value, threads corresponding to the M forward matching mask sequences are determined as the at least one thread, where M is a positive integer less than N; If none of the N forward matching mask sequences includes the first mask value, the last thread of the N threads is determined as the at least one thread.

5. The method according to claim 3, characterized in that: The determining the first position and the second position based on the first thread number and the hash table includes: Matching the data values ​​read by Q threads whose thread numbers are smaller than the first thread number among the N threads with the hash values ​​in the hash table, where Q is a positive integer; If the data values ​​read by L threads among the Q threads match the hash value in the hash table, the starting position of the data value read by the thread with the smallest thread number among the L threads is determined as the first position, where L is a positive integer less than or equal to Q; The starting position of the data value corresponding to the first hash value in the hash table is determined as the second position, wherein the first hash value is a hash value matching the data value read by the thread with the smallest thread number among the L threads.

6. The method according to claim 1, characterized in that The performing data compression based on the first position and the second position includes: In a thread parallel manner, a third position is determined based on the first position and a matching offset position corresponding to a j-th thread among the N threads, and a fourth position is determined based on the second position and the matching offset position, where the value range of j is [1, N]; Matching the data value at the third position with the data value at the fourth position to obtain a matching result corresponding to the j-th thread; Determine a first matching length based on the matching results corresponding to the N threads; Data compression is performed based on the first position, the second position and the first matching length.

7. The method according to claim 6, characterized in that The determining a first matching length based on the matching results corresponding to the N threads includes: If there is a position where the match fails in the matching results corresponding to the N threads, the data length between the fifth position and the sixth position is determined as the first matching length, the fifth position is the last position where the match fails before the first position, and the sixth position is the first position where the match fails after the first position; or, If the data values ​​in the matching results corresponding to the N threads are all matched, the matching offset positions corresponding to the N threads are updated based on the number of matching iterations, and the first matching length is determined based on the updated matching offset positions.

8. A data compression device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to read data through N threads in a thread parallel manner to obtain the first data, the data value read by each thread is the data value of K bytes in the first data, and N and K are both integers greater than 1; The acquisition module is further used to acquire N forward matching mask sequences, each of which is used to indicate a matching relationship between a data value read by a thread and a data value read by a thread that is previously sorted; The processing module is used to determine a first position and a second position based on the N forward matching mask sequences and the hash table corresponding to the first data, wherein the first position is the starting position of the first data value in the first data that is successfully forward matched, and the second position is the starting position of the data value that is before the first data value and matches the first data value; The processing module is further used to perform data compression based on the first position and the second position.

9. The device according to claim 8, characterized in that The acquisition module is specifically used to mark, in a thread parallel manner, threads that are sorted before the i-th thread and have the same data value as that read by the i-th thread as a first mask value, and mark threads that are sorted before the i-th thread and have different data values ​​read by the i-th thread as a second mask value, so as to obtain a forward matching mask sequence corresponding to the i-th thread; The value range of i is [2, N].

10. The device according to claim 8, characterized in that The processing module, the base body is used for: Determine at least one thread from the N threads based on the mask values ​​included in the N forward matching mask sequences; The first position and the second position are determined based on a first thread number and the hash table, wherein the first thread number is a thread number of a thread with a smallest thread number among the at least one thread.

11. The device according to claim 10, characterized in that The processing module is specifically used for: If M forward matching mask sequences among the N forward matching mask sequences include the first mask value, threads corresponding to the M forward matching mask sequences are determined as the at least one thread, where M is a positive integer less than N; If none of the N forward matching mask sequences includes the first mask value, the last thread of the N threads is determined as the at least one thread.

12. The device according to claim 10, characterized in that The processing module is specifically used for: Matching the data values ​​read by Q threads whose thread numbers are smaller than the first thread number among the N threads with the hash values ​​in the hash table, where Q is a positive integer; If the data values ​​read by L threads among the Q threads match the hash value in the hash table, the starting position of the data value read by the thread with the smallest thread number among the L threads is determined as the first position, where L is a positive integer less than or equal to Q; The starting position of the data value corresponding to the first hash value in the hash table is determined as the second position, wherein the first hash value is a hash value matching the data value read by the thread with the smallest thread number among the L threads.

13. The device according to claim 8, characterized in that The processing module is specifically used for: In a thread parallel manner, a third position is determined based on the first position and a matching offset position corresponding to a j-th thread among the N threads, and a fourth position is determined based on the second position and the matching offset position, where the value range of j is [1, N]; Matching the data value at the third position with the data value at the fourth position to obtain a matching result corresponding to the j-th thread; Determine a first matching length based on the matching results corresponding to the N threads; Data compression is performed based on the first position, the second position and the first matching length.

14. The device according to claim 13, characterized in that The processing module is specifically used for: If there is a position where the match fails in the matching results corresponding to the N threads, the data length between the fifth position and the sixth position is determined as the first matching length, the fifth position is the last position where the match fails before the first position, and the sixth position is the first position where the match fails after the first position; or, If the data values ​​in the matching results corresponding to the N threads are all matched, the matching offset positions corresponding to the N threads are updated based on the number of matching iterations, and the first matching length is determined based on the updated matching offset positions.

15. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the data compression method according to any one of claims 1 to 7 are implemented.

16. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the data compression method according to any one of claims 1 to 7 are implemented.

17. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the data compression method according to any one of claims 1 to 7.

Images