A data compression and decompression method and apparatus

Abstract

The application relates to the computer technical field, in particular to a data compression and decompression method and device. The data processing method comprises the following steps: obtaining an original data stream to be compressed; dividing the original data stream into several groups; processing each group of the original data stream to remove zero values and generate new data; the new data comprises an index and compressed data; the index is used for indicating whether the data at each position in each group of the original data stream is a zero value or a nonzero value; and the compressed data is the nonzero value data in the original data stream. The application can improve the compression rate, reduce the access bandwidth occupation when the system reads in data, and improve the computing capacity of the system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data compression and decompression method and apparatus. Background Technology

[0002] Data compression refers to a technical method that reduces the amount of data to decrease storage space and improve its transmission, storage, and processing efficiency without losing useful information, or reorganizes data according to certain algorithms to reduce data redundancy and storage space. Data compression can be divided into two types: lossless compression and lossy compression.

[0003] Lossless compression refers to reconstructing (or restoring, decompressing) the compressed data, ensuring that the reconstructed data is identical to the original. Lossless compression is used in situations where the reconstructed signal must be completely identical to the original signal. For example, lossless compression is used to compress disk files.

[0004] Lossy compression refers to reconstructing data from compressed data. The reconstructed data differs from the original data, but it does not affect the reader's understanding of the information conveyed by the original data and prevent misunderstanding. Lossy compression is suitable for situations where the reconstructed signal does not necessarily need to be completely identical to the original signal. For example, lossy compression can be used for image and audio compression. Summary of the Invention

[0005] In view of this, embodiments of this application provide a data compression and decompression method and apparatus, which can solve at least one technical problem in the related art.

[0006] In a first aspect, one embodiment of this application provides a data processing method, comprising: acquiring an original data stream to be compressed; dividing the original data stream into several groups; processing each group of the original data stream to remove zero values ​​and generate new data, the new data including an index and compressed data, wherein the index is used to indicate whether the data at each position in each group of the original data stream is a zero value or a non-zero value, and the compressed data is the non-zero value data in the original data stream.

[0007] In some embodiments, processing each group of the original data streams to remove zero values ​​and generate new data includes: traversing the data at each position in each group of the original data streams, generating an index corresponding to each group of the original data streams, and arranging the non-zero value data in each group of the original data streams after the index.

[0008] In some embodiments, the step of traversing the data at each position in each group of the original data stream and generating an index corresponding to each group of the original data stream includes: traversing the data at each position in each group of the original data stream; if it is determined that the data at the current position is zero-value data, then generating an index corresponding to the current position as first information; if it is determined that the data at the current position is non-zero-value data, then generating an index corresponding to the current position as second information.

[0009] In some embodiments, the method further includes recording information related to the total size of the original data stream.

[0010] In some embodiments, the method further includes: generating a head pointer, using the head pointer to read the first index, and determining whether the data at each position in the first group of raw data streams is zero-value data or non-zero-value data based on the first index, filling the zero-value data into the zero-value data position, moving the head pointer to read the non-zero-value data after the first index, and sequentially filling the non-zero-value data into the non-zero-value data positions of the first group of raw data streams until the first group of raw data streams is completely filled, thus completing the decompression of the first group of raw data; the head pointer reads the next index, and the same execution process is repeated for the previous index until all groups of raw data streams are completely filled, thus obtaining a complete raw data stream.

[0011] Secondly, one embodiment of this application provides a data decompression method, comprising: acquiring a compressed data stream, the compressed data stream including a plurality of indices and non-zero value data arranged after each index; each index and the non-zero value data arranged thereafter being used to decompress and generate a set of original data streams; decompressing and generating a plurality of sets of original data streams according to each of the plurality of indices and the non-zero value data arranged after that index, the plurality of sets of original data streams constituting an original data stream.

[0012] In some embodiments, the step of decompressing and generating several sets of raw data streams based on each of the several indices and the non-zero value data arranged after the index, wherein the several sets of raw data streams constitute the raw data stream, includes: determining whether the data at each position in each set of raw data streams is zero-value data or non-zero-value data based on each of the several indices; filling the zero-value data into the zero-value data position; and sequentially filling the non-zero value data arranged after the index into the non-zero value data position in order, thereby obtaining several sets of raw data streams, wherein the several sets of raw data streams constitute the raw data stream.

[0013] In some embodiments, the method further includes: accumulating the sum of the sizes of the decompressed data streams.

[0014] Thirdly, one embodiment of this application provides a data processing apparatus, comprising: a grouping module for acquiring an original data stream to be compressed and dividing the original data stream into several groups; and a compression module for processing each group of the original data stream to remove zero values ​​and generate new data, the new data including an index and compressed data, wherein the index is used to indicate whether the data at each position in each group of the original data stream is a zero value or a non-zero value, and the compressed data is the non-zero value data in the original data stream.

[0015] Fourthly, one embodiment of this application provides a data decompression apparatus, comprising: an acquisition module for acquiring compressed data, the compressed data including a plurality of indices and non-zero value data arranged after each index; each index and the non-zero value data arranged thereafter being used for decompression and generating a set of original data streams; and a decompression module for decompressing and generating a plurality of sets of original data streams according to each of the plurality of indices, and according to the index and the non-zero value data arranged thereafter, the plurality of sets of original data streams constituting an original data stream.

[0016] Fifthly, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a data processing method as described in any embodiment of the first aspect, or a data decompression method as described in any embodiment of the second aspect.

[0017] In a sixth aspect, one embodiment of this application provides a computer storage medium storing a computer program, which, when executed by a processor, implements the data processing method as described in any embodiment of the first aspect, or performs the data decompression method as described in any embodiment of the second aspect.

[0018] The beneficial effects of this application embodiment are as follows: by retaining the non-zero value data in the original data and deleting the zero value data, the compression rate is improved, the bandwidth occupied when the system reads data is reduced, and the computing power of the system is improved. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the implementation process of a data compression method provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the grouping of a raw data stream provided in one embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating an embodiment of the present application of processing each set of original data streams to remove zero values ​​and generate new data;

[0023] Figure 4 This is a schematic diagram illustrating a process of processing a set of raw data streams to remove zero values ​​and generate new data, provided in an embodiment of this application.

[0024] Figure 5 This application provides a schematic diagram of the implementation process of a data decompression method according to an embodiment;

[0025] Figure 6 This is a schematic diagram illustrating the storage of compressed data according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of a data decompression process provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of a data decompression process provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the logic circuit implementation process of a data decompression process provided in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram illustrating how original data streams of different lengths are compressed into the same data stream according to an embodiment of this application;

[0030] Figure 11 This is a schematic diagram illustrating the decompression of the same compressed data stream into original data streams of different lengths according to an embodiment of this application;

[0031] Figure 12A This is a schematic diagram of the structure of a data compression device provided in an embodiment of this application;

[0032] Figure 12B This is a schematic diagram of the structure of a data compression device provided in another embodiment of this application;

[0033] Figure 13A This is a schematic diagram of the structure of a data decompression device provided in an embodiment of this application;

[0034] Figure 13B This is a schematic diagram of the structure of a data decompression device provided in another embodiment of this application;

[0035] Figure 14This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0037] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0038] The terms "one embodiment" or "some embodiments" described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0039] Furthermore, in the description of this application, "a plurality of" means two or more. The terms "first" and "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] This application provides a data compression and decompression method and apparatus, which can compress and decompress data to reduce the original data size, thereby reducing the access bandwidth occupied when the system reads external data and improving the system's computing power.

[0041] To illustrate the technical solution of this application, specific embodiments are described below. It should be noted that the compression and decompression of convolutional neural network weight data are used as examples of specific application scenarios in the description of these embodiments. It should be understood that the data compression and decompression methods of this application can also be applied to other application scenarios; any application scenario requiring data compression and decompression is applicable to this application.

[0042] Currently, due to the complexity of synchronous processing and implementation of computational logic in convolutional neural networks (CNNs), compression of CNN weight data is generally not performed. Furthermore, even with sufficient computational performance, the performance of a CNN system is largely constrained by the access bandwidth of external memory, making it impossible to further increase computational power by adding more computing units. This application provides a data compression method and apparatus for compressing and decompressing CNN weight data, thereby reducing the size of the weight data, decreasing the bandwidth usage when reading weight data from external memory, and improving the system's computational power.

[0043] Let's first introduce the compression method for convolutional neural network weight data. When the weight data of a convolutional neural network includes "0" data, especially when there are many "0" values ​​in the weight data of a sparse convolutional neural network based on pruning, the weight data can be compressed.

[0044] Figure 1 The diagram shown is a schematic representation of the implementation flow of a data compression method according to an embodiment of this application. Figure 1 As shown, the data compression method includes steps S110 to S120.

[0045] S110: Obtain the raw data stream to be compressed and divide the raw data stream into several groups.

[0046] The original data stream to be compressed is the data stream before compression. The original data stream to be compressed is obtained and grouped into K groups according to a preset size, where K is a positive integer. The last group in the K groups is grouped according to the actual remaining size.

[0047] As one possible implementation method Figure 2 The diagram shows the grouping of the original data stream. For example... Figure 2 As shown, the preset size is M bytes. The original data stream is divided into K groups of M bytes each (Byte is a unit of data size, 1 Byte means 8 binary numbers, i.e. 1 Byte = 8 bits). The last group is divided into N bytes according to the actual remaining size. N and M are both positive integers, and N <= M.

[0048] S120, each set of original data is processed to remove zero values ​​and generate new data, including indexed and compressed data.

[0049] The index is used to indicate whether the data at each position in each set of original data streams is a zero value or a non-zero value. During the compression process, data with zero values ​​are removed and data with non-zero values ​​are retained. Therefore, the index can also be called a status indicator.

[0050] In one embodiment, the data at each position in each set of original data streams is traversed to generate an index corresponding to each set of original data streams. The non-zero value data in each set of original data streams is then arranged after the index, i.e., the index is placed at the beginning of each set of compressed data. The non-zero value data in each set of original data streams is arranged in order of its position after the index, thus preserving the original order of the non-zero value data in each set of original data streams.

[0051] As a non-restricted example, we iterate through the data at each position in each set of original data streams. If it is determined that the data at the current position is data to be deleted (e.g., the data at the current position is zero-value data), then we generate the index corresponding to that position as the first piece of information. If it is determined that the data at the current position is data to be retained (e.g., the data at the current position is non-zero-value data), then we generate the index corresponding to that position as the second piece of information. In other words, the index uses the first piece of information to identify that the original data at that position is data to be deleted, and uses the second piece of information to identify that the original data at that position is data to be retained.

[0052] As one possible implementation method Figure 3 The diagram illustrates how each set of original data streams is processed to generate new data. For example... Figure 3 As shown, each set of original data streams is processed to remove zero values ​​and generate new data. The new data includes an M-bit index Bitmap (bit is a unit of data size, 1 bit represents 1 binary number) and a reserved data (data) of any integer byte size less than or equal to M, i.e., compressed data. Each bit in the Bitmap is either 1 or 0 to indicate whether data at each position in each set of original data streams is deleted or retained. Specifically, if any byte in any set of original data is 0, the corresponding bit in the Bitmap for that byte is 0; if any byte in any set of original data is non-zero, the corresponding bit in the Bitmap for that byte is 1. Any byte in each set of original data is deleted if it is 0, and retained if it is not 0, and is sequentially arranged after the Bitmap. That is, the reserved data (data) in each set of original data is sequentially arranged after the corresponding Bitmap for that set of original data, preserving the order of the reserved data (data).

[0053] Figure 4 The diagram shown is a schematic representation of a process for processing a set of raw data streams to remove zero values ​​and generate new data, according to an embodiment. Figure 4 In the illustrated embodiment, M=16 is used as an example. Figure 4As shown, the raw data stream is represented in hexadecimal. One set of raw data consists of 16 bytes, with one cell representing 1 byte. The 16-byte raw data stream is as follows: 35, 00, 64, 3b, 00, 00, 00, 00, bb, 00, 73, 00, 00, 00, 00, 34. For each 16-byte raw data stream, its index Bitmap is represented in binary. Each bit in the Bitmap represents a one-to-one correspondence between the data of each byte in the raw data stream and whether it is zero. When the raw data of a byte is "00", the corresponding bit in the Bitmap is 0; when the byte data is not "00", the corresponding bit in the Bitmap is 1. A set of raw data consists of 16 bytes, and the Bitmap has 16 bits, with one cell representing 1 bit. The 16-bit Bitmap corresponding to the 16 bytes of data is as follows: 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1. Next, the 16 bits of the Bitmap are represented in hexadecimal as b0 and a1. Meanwhile, the non-zero values ​​in the original data stream are retained, resulting in the following compressed data: 35, 64, 3b, bb, 73, 34. This compressed data is then arranged sequentially after the hexadecimal Bitmap: b0, a1, 35, 64, 3b, bb, 73, 34. As you can see, after compression, the original 16 bytes of data have become 8 bytes.

[0054] It should be noted that in some possible implementations, for the last set of original data, when the set of original data is N bytes and does not satisfy (or is not equal to) M bytes, the Bitmap is padded with 0s if it is less than M bits.

[0055] The data compression method provided in this application embodiment can achieve a compression rate of approximately (m+10)% when the sparsity is m% (i.e., 1-m% of the data is 0). That is, the data becomes (m+10)% of the original size, which can reduce the bandwidth occupied when the system reads data and improve the system's computing power.

[0056] Corresponding to the data compression method described above, the data decompression method will be introduced next. For details not described in the data decompression method section, please refer to the data compression method section.

[0057] Figure 5 The diagram shown is a schematic representation of the implementation flow of a data decompression method according to an embodiment of this application. Figure 5 As shown, the data decompression method includes steps S210 to S220.

[0058] S210, Obtain the compressed data stream, which includes several indices and non-zero value data arranged after each index; each index and the non-zero value data arranged thereafter are used to decompress and generate a set of original data streams.

[0059] Each index is used to indicate whether the original data at each position in a set of original data streams is zero-value data or non-zero-value data.

[0060] It should be noted that if a set of original data streams does not include non-zero value data, then no non-zero value data will be arranged after the index corresponding to that set of original data streams.

[0061] S220, based on each of the several indices and the non-zero value data arranged after that index, decompress and generate several sets of raw data streams, and the several sets of raw data streams form the raw data stream.

[0062] In one embodiment, several sets of raw data streams are obtained based on each index in a plurality of indices and the non-zero value data arranged after each index, and the plurality of sets of raw data streams constitute the raw data stream.

[0063] In one embodiment, based on each of several indices, it is determined whether the data at each position in each set of original data streams is zero-value data or non-zero-value data. Zero-value data is filled into the position of zero-value data, and several non-zero-value data arranged after the index are sequentially filled into the position of non-zero-value data to obtain several sets of original data streams. The several sets of original data streams constitute the original data stream.

[0064] In one embodiment, several groups of raw data streams are sequentially arranged according to a number of indices to generate a raw data stream.

[0065] In one embodiment, a head pointer is generated, and the first index is read using the head pointer. Based on the first index, it is determined whether the data at each position in the first group of original data streams is zero-value data or non-zero-value data. Zero-value data is filled into the zero-value data positions. The head pointer is moved to read the non-zero-value data after the first index, and the non-zero-value data is sequentially filled into the non-zero-value data positions of the first group of original data streams until the first group of original data streams is completely filled, completing the decompression of the first group of original data. The head pointer then reads the next index, repeating the same execution process for the previous index, until all groups of original data streams are completely filled, thus obtaining a complete original data stream. Preferably, the index is converted according to the amount of data in the first group of original data streams, restoring the index to the M-byte data corresponding to the original data stream.

[0066] Continuing from the above Figure 4The implementation shown, as one possible approach, sets up a set of data stream buffers to store the compressed data stream read from external memory, and generates a head pointer `head_index`. This implementation typically reconstructs the compressed data based on the number of data items in each group when the original data was grouped; specifically, M=16 is used as an example. Figure 6 The diagram shows a schematic of the storage of the compressed data. Figure 7 and Figure 8 The diagram shown illustrates the data decompression process. Figure 9 for Figure 7 and Figure 8 The diagram shows the logic circuit implementation flow of the data decompression process. It should be noted that due to image size limitations... Figure 9 Only shown Figures 6 to 8 The data shown is a portion of the compressed data.

[0067] Combination Figure 6 , Figure 7 , Figure 8 and Figure 9 The following details the decompression process of the first 16-byte data stream. First, the index and the non-zero data following the index are located based on the `head_index`. The data is then decompressed to restore the original data stream. The index indicated by `head_index` is extracted; it is a 16-bit data represented in hexadecimal as b0 and a1. The data at the index is then converted back to the corresponding 16-byte data. First, the 16-bit data is represented in binary as: 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1. The binary data is then converted back to the corresponding 16 bytes, as shown below. Figure 7 As shown, if the data at a certain position is 0, then 0 0 is written to that position; if the data at a certain position is 1, then that position is left unfilled. This reconstructs a template of the original data stream, allowing us to determine whether each position in a 16-byte original data set is zero or non-zero. Furthermore, as... Figure 8 As shown, the non-zero values ​​arranged after the index are sequentially filled into the empty parts (i.e., the bytes without "00") of the restored original data stream template. Simultaneously, for each byte of data filled, the head_index moves forward by 1 byte (8 bits). This completes the decompression process of a set of 16 bytes of data, and the head_index points to the starting position of the next set of data to be decompressed. The decompression process of the first set of 16-byte data streams is repeated to complete the decompression of K-1 sets of 16-byte data streams.

[0068] It should be noted that the decompression of the Kth group, which is the last group of data streams (N bytes in size, N<=M), requires special handling. The following explanation will also use M=16.

[0069] As can be seen from the compression principle of the embodiments of this application, when a set of original data streams is less than M = 16 bytes, original data streams of different sizes may become the same data after compression. Figure 10 The diagram illustrates how raw data streams of different sizes are compressed into the same data. Figure 10 As shown, there are four hexadecimal data streams of different sizes, numbered 16 bytes, 12 bytes, 6 bytes, and 4 bytes from top to bottom. After compression, these four hexadecimal data streams become the same hexadecimal data stream: b0, 00, 35, 64, 3b. Therefore, additional information is needed during decompression to restore the exact size of the original data stream. Optionally, this additional information may include information related to the size of the original data stream.

[0070] In one embodiment of this application, the data compression method further includes: recording the total size of the original data stream. Correspondingly, the data decompression method further includes: accumulating the sum of the recorded sizes of the decompressed data streams. In this embodiment, by comparing the accumulated sum of the recorded sizes of the decompressed data streams with the recorded total size of the original data stream, it can be confirmed whether the decompression of all original data streams has been completed, and / or, the size N of the last set of decompressed data streams can be confirmed, thereby obtaining a set of original data streams of size N when decompressing data for the last index.

[0071] In another embodiment of this application, the data compression method further includes: recording the value obtained by taking the total size of the original data stream modulo a preset size. For example, if the total size of the original data stream is Num Byte and the preset size is M Byte, the remainder of Num divided by M is denoted as Num%M.

[0072] As one possible implementation, the method of recording the total size of the original data stream modulo M=16 is used. The value of the total size of the original data stream modulo M=16 is TAIL_MOD. The corresponding length of the last group of original data streams is determined based on the value of TAIL_MOD, so that the decompression process of the last group of original data streams can be completed accurately.

[0073] As a non-restrictive example, Figure 11 The diagram illustrates the decompression of the same compressed data stream into original data streams of different lengths. Figure 11As shown, the compressed hexadecimal data stream is: b0, 00, 35, 64, 3b. When the TAIL_MOD value in decimal is 0, 12, 6, and 4 respectively, the lengths of the original decompressed data streams are 16 bytes, 12 bytes, 6 bytes, and 4 bytes respectively.

[0074] In some embodiments, the decompression process is performed in parallel, and the decompression of at least M-byte data streams can be completed in one clock cycle.

[0075] The decompression method used in one embodiment of this application can employ parallel operation of logic circuits, decompressing n*M bytes of data in one clock cycle, where n is a positive integer greater than 0. This can meet the high bandwidth requirements of convolutional neural network weight data, and increasing n can flexibly achieve even greater data bandwidth.

[0076] An embodiment of this application also provides a data processing method, including at least some steps of the data compression method and / or the data decompression method described in any of the foregoing embodiments.

[0077] In one embodiment, the data processing method includes:

[0078] S110: Obtain the raw data stream to be compressed and divide the raw data stream into several groups.

[0079] S120, each set of original data is processed to remove zero values ​​and generate new data, including indexed and compressed data.

[0080] S220, based on each of the several indices and the non-zero value data arranged after that index, decompress and generate several sets of raw data streams, and the several sets of raw data streams form the raw data stream.

[0081] In one embodiment, a head pointer is generated, the first index is read using the head pointer, and the data at each position in the first group of raw data streams is determined to be zero-value data or non-zero-value data based on the first index. Zero-value data is filled into the zero-value data position. The head pointer is moved to read the non-zero-value data after the first index, and the non-zero-value data is filled into the non-zero-value data positions of the first group of raw data streams in turn until the first group of raw data streams is completely filled, thus completing the decompression of the first group of raw data. The head pointer reads the next index, and the same execution process is repeated for the previous index until all groups of raw data streams are completely filled, thus obtaining a complete raw data stream.

[0082] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0083] An embodiment of this application also provides a data compression apparatus. For details not described in detail in this data compression apparatus, please refer to the relevant descriptions in the foregoing data compression method embodiments.

[0084] See Figure 12A , Figure 12A This is a schematic diagram of the structure of a data compression device provided in an embodiment of this application. Figure 12A As shown, the data compression device may include a grouping module 1201 and a compression module 1202.

[0085] The grouping module 1201 is used to acquire the raw data stream to be compressed and divide the raw data stream into several groups.

[0086] Compression module 1202 is used to process each set of original data streams to remove zero values ​​and generate new data. The new data includes index and compressed data. The index is used to indicate whether the data at each position in each set of original data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the original data stream.

[0087] In some embodiments, Figure 12A Based on the illustrated embodiments, as Figure 12B As shown, the data compression device also includes a first recording module 1203.

[0088] The first recording module 1203 is used to record information related to the total size of the original data.

[0089] In some embodiments, information related to the total size of the original data includes: the total size of the original data, the size of the last set of original data, or the value of the total size of the original data modulo a preset size.

[0090] One embodiment of this application also provides a data decompression apparatus. For details not described in the foregoing embodiments of the data decompression method, please refer to the relevant descriptions of the data decompression apparatus.

[0091] See Figure 13A , Figure 13A This is a schematic diagram of a data decompression apparatus according to an embodiment of this application. The data decompression apparatus may include: an acquisition module 1301 and a decompression module 1302.

[0092] The acquisition module 1301 is used to acquire the compressed data stream, which includes several indices and non-zero value data arranged after each index; each index and the non-zero value data arranged after it are used to decompress and generate a set of original data streams.

[0093] The decompression module 1302 is used to decompress and generate several sets of raw data streams based on each of the several indices, the index and the non-zero value data arranged after the index, and the several sets of raw data streams constitute the raw data stream.

[0094] In some embodiments, the decompression module 1302 is specifically used to generate a head pointer, read the first index using the head pointer, and determine whether the data at each position in the first group of original data streams is zero-value data or non-zero-value data based on the first index. Zero-value data is filled into the zero-value data position, and the head pointer is moved to read the non-zero-value data after the first index. Non-zero-value data is then filled into the non-zero-value data positions of the first group of original data streams in sequence until the first group of original data streams is completely filled, thus completing the decompression of the first group of original data. The head pointer reads the next index, and the same execution process is repeated for the previous index until all groups of original data streams are completely filled, thus obtaining a complete original data stream.

[0095] In some embodiments, Figure 13A Based on the illustrated embodiments, as Figure 13B As shown, the data decompression device also includes a second recording module 1303.

[0096] In one embodiment, the second recording module 1303 is used to accumulate and record the sum of the sizes of the decompressed data during the decompression process.

[0097] Specifically, the second recording module 1303 is used to compare the sum of the sizes of the accumulated decompressed data with the total size of the original data until the sum of the sizes of the accumulated decompressed data equals the total size of the original data, at which point the decompression module 1302 is triggered to stop decompression, and the decompression module 1302 obtains the complete original data.

[0098] In one embodiment, the second recording module 1303 is used to record the size of the last set of decompressed data. When the size of the last set of decompressed data meets the size of the last set of data before compression, the decompression module is triggered to stop decompression, and the decompression module obtains the complete original data.

[0099] An embodiment of this application also provides a data processing apparatus, including at least some modules of the aforementioned data compression apparatus and / or data decompression apparatus.

[0100] In one embodiment, the data processing device includes a grouping module 1201, a compression module 1202, and a decompression module 1302.

[0101] The grouping module 1201 is used to acquire the raw data stream to be compressed and divide the raw data stream into several groups.

[0102] Compression module 1202 is used to process each set of original data streams to remove zero values ​​and generate new data. The new data includes index and compressed data. The index is used to indicate whether the data at each position in each set of original data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the original data stream.

[0103] The decompression module 1302 is used to generate a head pointer, read the first index using the head pointer, and determine whether the data at each position in the first group of raw data streams is zero-value data or non-zero-value data based on the first index. Zero-value data is filled into the zero-value data position. The head pointer is moved to read the non-zero-value data after the first index, and the non-zero-value data is filled into the non-zero-value data positions of the first group of raw data streams in sequence until the first group of raw data streams is completely filled, thus completing the decompression of the first group of raw data. The head pointer reads the next index, and the same execution process is repeated for the previous index until all groups of raw data streams are completely filled, thus obtaining a complete raw data stream.

[0104] One embodiment of this application also provides an electronic device, such as... Figure 14 As shown, the electronic device may include one or more processors 1400. Figure 14 (Only one is shown in the diagram), a memory 1410, and a computer program 1420 stored in the memory 1410 and executable on one or more processors 1400, such as a data compression and / or data decompression program. When one or more processors 1400 execute the computer program 1420, they can implement the various steps in the data compression method and / or data decompression method embodiments. Alternatively, when one or more processors 1400 execute the computer program 1420, they can implement the functions of various modules / units in the data compression apparatus and / or data decompression apparatus embodiments, without limitation herein.

[0105] For example, computer program 1420 may be divided into one or more modules / units, one or more of which are stored in memory 1410 and executed by processor 1400 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 1420 in the processing unit.

[0106] For example, computer program 1420 can be divided into the following modules. The specific functions of each module are as follows:

[0107] The grouping module is used to acquire the raw data stream to be compressed and divide the raw data stream into several groups.

[0108] The compression module processes each set of raw data streams to remove zero values ​​and generate new data. The new data includes an index and compressed data. The index indicates whether the data at each position in each set of raw data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the raw data stream.

[0109] For example, computer program 1420 can be divided into the following modules. The specific functions of each module are as follows:

[0110] The acquisition module is used to acquire compressed data, which includes several indices and non-zero value data arranged after each index; each index and the non-zero value data arranged after it are used to decompress and generate a set of raw data streams;

[0111] The decompression module is used to decompress and generate several sets of raw data streams based on each of several indices, the index and the non-zero value data arranged after the index, and the several sets of raw data streams constitute the raw data stream.

[0112] Those skilled in the art will understand that Figure 14 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. Electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different components. For example, electronic devices may also include input / output devices, network access devices, buses, etc.

[0113] In one embodiment, the processor 1400 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] In one embodiment, memory 1410 may be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory 1410 may also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 1410 may include both internal and external storage units. Memory 1410 is used to store computer programs and other programs and data required by the electronic device. Memory 1410 may also be used to temporarily store data that has been output or will be output.

[0115] This application also provides another preferred embodiment of an electronic device, in which the electronic device includes one or more processors. The one or more processors are used to execute the following program modules stored in memory:

[0116] The grouping module is used to acquire the raw data stream to be compressed and divide the raw data stream into several groups.

[0117] The compression module processes each set of raw data streams to remove zero values ​​and generate new data. The new data includes an index and compressed data. The index indicates whether the data at each position in each set of raw data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the raw data stream.

[0118] And / or,

[0119] The acquisition module is used to acquire compressed data, which includes several indices and non-zero value data arranged after each index; each index and the non-zero value data arranged after it are used to decompress and generate a set of raw data streams;

[0120] The decompression module is used to decompress and generate several sets of raw data streams based on each of several indices, the index and the non-zero value data arranged after the index, and the several sets of raw data streams constitute the raw data stream.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0122] An embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the various steps in the embodiments of the data compression method and / or data decompression method and / or data processing method.

[0123] An embodiment of this application also provides a computer program product that, when run on an electronic device, enables the electronic device to implement the various steps in the embodiments of the data compression method and / or data decompression method and / or data processing method.

[0124] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0130] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the raw data stream to be compressed, and divide the raw data stream into K groups according to a preset size M bytes, where K is a positive integer. The Kth group is divided according to the actual remaining size N bytes, where N and M are both positive integers and N≤M. Each group of the original data streams is processed to remove zero values ​​and generate new data. The new data includes an index and compressed data. The index is a fixed-length bitmap that corresponds one-to-one with the byte position in the corresponding group of the original data streams. Each bit in the bitmap is used to indicate whether the data at each position in each group of the original data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the original data streams. Traverse the data at each position in each group of the original data stream, generate the bitmap corresponding to the index of each group of the original data stream, and arrange the zero-value data in each group of the original data stream in the order of their order in the original data stream after the bitmap of the index, forming a compression unit consisting of the bitmap and the non-zero-value data immediately following the bitmap; the compression units corresponding to each group are arranged continuously in the grouping order of several groups to form the compressed data stream; record information related to the total size of the original data stream, or record the actual size N of the last group of the original data stream.

2. The data processing method as described in claim 1, characterized in that, The step of traversing the data at each position in each group of the original data stream and generating an index corresponding to each group of the original data stream includes: Traverse the data at each position in each group of the original data stream. If the data at the current position is determined to be zero value data, generate the index corresponding to the current position as first information; if the data at the current position is determined to be non-zero value data, generate the index corresponding to the current position as second information.

3. The data processing method according to any one of claims 1 to 2, characterized in that, Also includes: Record information related to the total size of the original data stream.

4. The data processing method according to any one of claims 1 to 2, characterized in that, Also includes: A head pointer is generated. The first index is read using the head pointer, and the data at each position in the first group of raw data streams is determined to be zero or non-zero based on the first index. Zero-value data is filled into the zero-value data positions. The head pointer is moved to read the non-zero-value data after the first index, and the non-zero-value data is filled into the non-zero-value data positions of the first group of raw data streams in sequence until the first group of raw data streams is completely filled, thus completing the decompression of the first group of raw data streams. The head pointer reads the next index, and the same execution process is repeated for the previous index until all groups of raw data streams are completely filled, thus obtaining a complete raw data stream.

5. A data decompression method, characterized in that, include: Obtain a compressed data stream, the compressed data stream including several indices and non-zero value data arranged after each index; Each index and the non-zero value data following it are used to decompress and generate a set of raw data streams; The index is a fixed-length bitmap that corresponds one-to-one with the position of each byte in the corresponding group of original data streams. Each bit in the bitmap is used to indicate whether the data at each position in each group of original data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the original data stream. The bitmap in the current compression unit is read using the head pointer, and the data at each position in the current group of original data streams is determined to be a zero value or a non-zero value based on the bitmap. Fill the zero value into the zero value position, and read the non-zero value data immediately following the bitmap in sequence, and fill it into the non-zero value position in sequence to complete the decompression of the original data stream of the current group; The moving head pointer reads the bitmap in the next compression unit until all groups of original data streams are completely restored; The compressed data stream consists of several compression units arranged consecutively in the original grouping order. Each compression unit consists of a bitmap and non-zero value data immediately following the bitmap. Based on each of the several indices and the non-zero value data arranged after that index, decompression is performed to generate several sets of raw data streams, and the several sets of raw data streams constitute the raw data stream; The actual length of the last set of raw data streams is determined based on information related to the total size of the recorded raw data streams, or the actual size of the last set of raw data streams recorded.

6. The data decompression method as described in claim 5, characterized in that, Also includes: The sum of the sizes of the decompressed data streams is recorded, or the size of the last set of original data streams after decompression is recorded.

7. A data processing apparatus, characterized in that, include: The grouping module is used to acquire the raw data stream to be compressed, and divide the raw data stream into K groups according to a preset size M bytes, where K is a positive integer. The Kth group is divided according to the actual remaining size N bytes, where N and M are both positive integers and N≤M. A compression module is used to process each group of the original data streams to remove zero values ​​and generate new data. The new data includes an index and compressed data. The index is a fixed-length bitmap that corresponds one-to-one with the byte positions in the corresponding group of original data streams. Each bit in the bitmap is used to indicate whether the data at each position in each group of the original data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the original data streams. Traverse the data at each position in each group of the original data stream, generate the bitmap corresponding to the index of each group of the original data stream, and arrange the zero-value data in each group of the original data stream in the order of their order in the original data stream after the bitmap of the index, forming a compression unit consisting of the bitmap and the non-zero-value data immediately following the bitmap; the compression units corresponding to each group are arranged continuously in the grouping order of several groups to form the compressed data stream; record information related to the total size of each group of the original data stream, or record the actual size N of the last group of the original data stream.

8. A data decompression apparatus, characterized in that, include: The acquisition module is used to acquire compressed data, which includes several indices and non-zero value data arranged after each index. Each index and the non-zero value data arranged after it are used to decompress and generate a set of original data streams. Each index is a fixed-length bitmap that corresponds one-to-one with the position of each byte in the set of original data streams. Each bit in the bitmap is used to indicate whether the data at each position in each set of original data streams is a zero value or a non-zero value. The compressed data is the non-zero value data in the original data streams. The bitmap in the current compression unit is read using the head pointer, and the data at each position in the current set of original data streams is determined to be a zero value or a non-zero value based on the bitmap. Fill the zero value into the zero value position, and read the non-zero value data immediately following the bitmap in sequence, and fill it into the non-zero value position in sequence to complete the decompression of the original data stream of the current group; The moving head pointer reads the bitmap in the next compression unit until all groups of original data streams are completely restored; The compressed data stream consists of several compression units arranged consecutively in the original grouping order. Each compression unit consists of a bitmap and non-zero value data immediately following the bitmap. The decompression module is used to decompress and generate several sets of raw data streams based on each of the several indices, the index and the non-zero value data arranged after the index, the several sets of raw data streams forming a raw data stream; and to determine the actual length of the last set of raw data streams based on information related to the total size of the recorded raw data streams, or the actual size of the last set of raw data streams recorded.

Images