Compression method, compression system, chip, electronic equipment and storage medium

Abstract

The invention provides a compression method, a compression system, a chip, electronic equipment and a storage medium, and the method comprises the steps: setting a plurality of compression dictionaries for a plurality of hash cores in a scene of carrying out data compression based on the compression dictionaries, carrying out the parallel query of the hash values of first data corresponding to the plurality of hash cores for the corresponding plurality of compression dictionaries, according to the data compression method and device, the conflict problem of single compression dictionary query of multiple hash cores is reduced, meanwhile, the corresponding compression dictionary is queried in parallel through multiple hash values, synchronous processing of multiple pieces of first data is achieved, and the data compression efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular, to a compression method, a compression system, a chip, an electronic device, and a storage medium. Background Art

[0002] Due to the huge progress of modern information technology, the amount of data to be processed has increased rapidly. This has brought great pressure to both the storage and transmission of information. Therefore, it is necessary to compress data to reduce the pressure of storage and transmission.

[0003] In the related art, lossless compression is usually used for general data compression. When compressing based on a dictionary model compression algorithm, there is a problem of low compression efficiency. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems in the related art to some extent.

[0005] To this end, the present application provides a compression method, a compression system, a chip, an electronic device, and a storage medium to improve the compression efficiency.

[0006] An embodiment of one aspect of the present application provides a compression method, including:

[0007] Determining hash values obtained by each hash core in a compression engine for performing hash processing on first data to be processed;

[0008] Parallelly querying corresponding compression dictionaries according to the hash values of multiple first data, and determining matching data of the hash values of each first data in the corresponding compression dictionaries;

[0009] Determining a compression sequence corresponding to each first data according to the matching data of the hash values of the multiple first data in the corresponding compression dictionaries.

[0010] An embodiment of another aspect of the present application provides a compression system, including: a compression module;

[0011] The compression module is configured to determine hash values obtained by each hash core in a compression engine for performing hash processing on first data to be processed, parallelly query corresponding compression dictionaries according to the hash values of multiple first data, determine matching data of the hash values of each first data in the corresponding compression dictionaries, and determine a compression sequence corresponding to each first data according to the matching data of the hash values of the multiple first data in the corresponding compression dictionaries.

[0012] An embodiment of another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the foregoing aspect is implemented.

[0013] In another embodiment of the present application, a chip is provided, characterized in that the chip includes a processing circuit configured to implement the method described in the foregoing one aspect.

[0014] In another embodiment of the present application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the foregoing one aspect is implemented.

[0015] In another embodiment of the present application, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in the foregoing one aspect is implemented.

[0016] For the compression method, compression system, chip, electronic device and storage medium provided by the present application, in the scenario of data compression based on a compression dictionary, multiple compression dictionaries are set for multiple hash cores, and the hash values of the first data corresponding to the multiple hash cores are queried in parallel for the corresponding multiple compression dictionaries, reducing the conflict problem of a single compression dictionary query for multiple hash cores. At the same time, the multiple hash values are queried in parallel for the corresponding compression dictionaries to synchronously process multiple first data, improving the efficiency of data compression.

[0017] Some of the additional aspects and advantages of the present application will be given in the following description, some will become obvious from the following description, or will be understood through the practice of the present application. Description of the Drawings

[0018] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0019] Figure 1 It is a schematic flowchart of a compression method provided by an embodiment of the present application;

[0020] Figure 2 It is a schematic flowchart of another compression method provided by an embodiment of the present application;

[0021] Figure 3 It is one of the schematic diagrams of a dictionary matrix provided by an embodiment of the present application;

[0022] Figure 4 It is another schematic diagram of a dictionary matrix provided by an embodiment of the present application;

[0023] Figure 5A It is a schematic flowchart of another compression method provided by an embodiment of the present application;

[0024] Figure 5B It is a schematic diagram of a compression query provided by an embodiment of the present application;

[0025] Figure 6 It is a schematic flowchart of another compression method provided by an embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of the position offset provided by an embodiment of the present application;

[0027] Figure 8 It is a schematic structural diagram of a compression system provided by an embodiment of the present application;

[0028] Figure 9 It is a schematic structural diagram of another compression system provided by an embodiment of the present application;

[0029] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0030] Figure 11 It is a schematic structural diagram of a chip proposed by an embodiment of the present application. Detailed implementation manners

[0031] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0032] In the method based on dictionary compression, a compression dictionary and a compression engine are provided in the compression module, and a hash kernel is provided in the compression engine. The data to be compressed is hashed by the hash kernel to obtain a hash value, and based on the hash value, the compression dictionary is queried to determine whether the data to be compressed can be compressed. In the related art, in the case of multiple hash kernels, the multiple hash kernels need to access the compression dictionary serially, and the processing efficiency is low. If the multiple hash kernels access the compression dictionary in parallel, there will be conflicts, resulting in the inability to improve the compression efficiency.

[0033] Therefore, the embodiments of the present application propose a compression method, a compression system, a chip, an electronic device and a storage medium. In the scenario of data compression based on a compression dictionary, multiple compression dictionaries are set for multiple hash kernels, and the hash values of the first data corresponding to the multiple hash kernels are queried in parallel for the corresponding multiple compression dictionaries, reducing the conflict problem of multiple hash kernels querying a single compression dictionary. At the same time, the multiple hash values are queried in parallel for the corresponding compression dictionaries to synchronously process multiple first data, improving the efficiency of data compression.

[0034] The compression method, compression system, chip, electronic device and storage medium of the embodiments of the present application will be described below with reference to the accompanying drawings.

[0035] Figure 1 A flowchart of a compression method provided by an embodiment of the present application.

[0036] In the embodiment of the present application, an example is given in which the compression method is configured in a compression module. The compression module can be applied to any electronic device or chip, so that the electronic device can perform data processing functions.

[0037] Among them, the electronic device can be any device with computing power. For example, it can be a mobile terminal, and the mobile terminal can be a hardware device such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc. with various operating systems, touch screens, and / or display screens.

[0038] Among them, the chip includes an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an embedded system chip, a system on a chip (SOC), a reduced instruction set computer (RISC), etc., which will not be listed one by one here.

[0039] As Figure 1 shown, the method may include the following steps:

[0040] Step 101: Determine the hash values obtained by performing hash processing on the first data to be processed by each hash core in the compression engine.

[0041] In the embodiment of the present application, the first data to be processed by each hash core in the compression engine is determined from the data to be compressed. Among them, the data to be compressed can be compressed in units of pages, or in units of data blocks of a set size, or in units of frames. Thus, the data to be compressed can be a page of data, or a block of data, or a frame of data, or the sum of unit data, which is not limited in this embodiment. The data to be compressed can be loaded into a cache unit for storing the data to be compressed, which can be called a compression input data buffer.

[0042] In the embodiment of the present application, the compression engine is used to execute a compression command to identify and compress the data to be compressed. The compression engine includes multiple hash cores. The compression engine determines the first data to be processed by each hash core from the data to be compressed, and realizes parallel processing of the data to be compressed through multiple hash cores to obtain multiple hash results.

[0043] Among them, as an implementation method, according to the first starting position and the position offset of reading data from the data to be compressed currently, determine the starting positions of each hash core to read data from the data to be compressed. According to the starting positions of each hash core to read data and the amount of data to be processed, determine the first data to be processed by each hash core. As an example, the data to be compressed is in units of pages, initialize the offset address in the page, and the offset address can be indicated by the value of a pointer. For example, Pointer = 0. According to the rules of the compression algorithm and the number of hash cores, appropriate data is taken from the position indicated by Pointer = 0 in the page for hash processing. For example, the compression algorithm is LZ4, and there are a total of 4 hash cores. In the LZ4 algorithm, each hash core processes 1 word at a time, that is, 4 bytes. Therefore, 4 hash cores can process 4 words of data in parallel, and the starting position of each hash core to process data is offset backward by 1 byte. Thus, 7 bytes of data in the data to be compressed can be processed at a time. Compared with the case of only one hash core, the amount of data processed synchronously is increased, and the compression efficiency is improved. It should be noted that each time multiple hash cores obtain data from the data to be compressed, there is a starting position. Then, the multiple hash cores perform data reading after offsetting based on this starting position to implement obtaining the corresponding data to be processed by each hash core each time.

[0044] In the embodiments of the present application, each hash core performs hash processing on the first data to be processed in parallel to obtain the hash values of each first data. Among them, the hash values of each first data include the characteristic information of the first data. Using the hash values to query the compression dictionary reduces the data storage amount in the compression dictionary and also improves the query efficiency.

[0045] Step 102, query the corresponding compression dictionary in parallel according to the hash values of multiple first data, and determine the matching data of the hash values of each first data in the corresponding compression dictionary.

[0046] In the first implementation method of the embodiments of the present application, the access port of the compression dictionary is one. In order to reduce the problem of access conflicts, there are multiple compression dictionaries in the compression engine, and the number of compression dictionaries is greater than or equal to the number of hash cores. That is to say, the hash values of the first data processed by each hash core query the compression dictionaries corresponding to each hash core through the access port of the compression dictionary, so as to realize the parallel access of multiple hash cores to multiple compression dictionaries, improve the compression performance and avoid the conflict problem of multiple hash cores accessing the same compression dictionary.

[0047] In the second implementation manner of the embodiment of the present application, there are multiple access ports for the compression dictionary, and there is at least one dictionary in the compression engine. The number of access ports included in at least one compression dictionary is greater than or equal to the number of hash kernels. That is to say, the hash value of the first data processed by each hash kernel is used to query the compression dictionary through the corresponding access port, so as to realize the parallel access of multiple hash kernels to the compression dictionary, improve the compression performance, and avoid the conflict problem of multiple hash kernels accessing the same compression dictionary. It should be noted that, in order to improve the compression speed, multiple hash kernels are set. The data segments (first data) processed by multiple hash kernels are not completely the same. Setting multiple hash kernels will reduce the amount of data in the compression dictionary compared with the compression dictionary corresponding to one hash kernel, resulting in a loss of compression ratio. Therefore, one hash kernel can correspond to multiple compression dictionaries to reduce the loss of compression ratio and further improve the compression performance.

[0048] In the embodiment of the present application, by querying the corresponding compression dictionary in parallel with the hash values of multiple first data, the matching data of the hash values of each first data in the compression dictionary can be quickly determined. The matching data includes the hash value and the position information. The hash value and the position information matching each first data can be stored in the form of key-value pairs. Among them, the position information corresponding to the hash value matching each first data is used to indicate the storage position of the data that may be the same as each first data.

[0049] Step 103: Determine the compression sequence corresponding to each first data according to the matching data of the hash values of multiple first data in the corresponding compression dictionary.

[0050] In the embodiment of the present application, according to the matching data of the hash values of multiple first data in the corresponding compression dictionary, it can be identified whether each first data is the data that first appears in the corresponding compression dictionary, that is, whether it is the data that first appears in the data to be compressed. The following is described for different scenarios:

[0051] In one scenario, if it is the data that first appears, no compression process is performed, and encoding is performed according to the requirements of the compression algorithm to obtain the corresponding compression sequence.

[0052] In another scenario, if it is not the data that first appears, but the data that appears repeatedly, then according to the position information of this first data in the data to be compressed and the position information of the first data that has appeared historically, a backward traversal method is used for matching to determine the longest matching data, and the longest matching data is used as the data to be compressed for compression to obtain the output compression sequence.

[0053] In the compression method of the embodiment of the present application, the first data to be processed by each hash core is determined from the data to be compressed, each hash core performs hash processing on the first data to be processed to obtain the hash values of each first data, and the corresponding compression dictionary is queried in parallel according to the hash values of multiple first data to determine the matching data of the hash values of each first data in the corresponding compression dictionary. According to the matching data of the hash values of multiple first data in the corresponding compression dictionary, the compression sequence corresponding to each first data is determined. In the scenario of multiple hash cores, multiple compression dictionaries are set to correspond to multiple hash cores, and multiple hash values are used to query the corresponding compression dictionary in parallel, which improves the processing efficiency of the data to be compressed, thereby improving the data compression efficiency.

[0054] Based on the above embodiment, Figure 2 is a schematic flowchart of another compression method provided by the embodiment of the present application, as Figure 2 shown, the method includes the following steps:

[0055] Step 201, determine the hash values obtained by each hash core in the compression engine performing hash processing on the first data to be processed.

[0056] Among them, step 201 can refer to the relevant explanations in the foregoing embodiment, with the same principle, which will not be elaborated here.

[0057] Step 202, according to the corresponding relationship between each hash core and the compression dictionary stored in the dictionary matrix, determine the compression dictionary corresponding to each hash core from the dictionary matrix.

[0058] Among them, multiple compression dictionaries are stored in multiple different vectors in the dictionary matrix, and the vector is a row vector or a column vector.

[0059] In the embodiment of the present application, multiple identical compression dictionaries are stored in the dictionary matrix. The dictionary matrix is a memory array, which is composed of multiple identical single-port RAMs or pseudo-dual-port RAMs. That is to say, the data stored in the array organized by multiple single-port RAMs (Single-Port RAM) constitutes multiple compression dictionaries of the present application. Each hash core has a corresponding compression dictionary. That is to say, the compression engine includes compression dictionaries corresponding to multiple hash cores, and the data in the compression dictionaries corresponding to each hash core is the same. The first data corresponding to multiple hash cores avoids access conflicts by accessing the compression dictionary composed of multiple RAMs in the compression dictionary in parallel, and improves data throughput and query efficiency.

[0060] As an implementation, the dictionary matrix consists of (X + 1) * (Y + 1) RAMs. Each RAM is an element in the dictionary matrix and is used to store the data of the compressed dictionary. The RAM can be a single-port RAM or a pseudo dual-port RAM. That is, the dictionary matrix includes (X + 1) row vectors and (Y + 1) column vectors. Among them, the column vector is a (X + 1) × 1 matrix, where (X + 1) is the number of elements in this column, and the row vector is a 1 × (Y + 1) matrix, where (Y + 1) is the number of elements in this row. And the number of elements included in each compressed dictionary is the same as the number of hash kernels to achieve the subsequent synchronous update of the compressed dictionary. For the storage methods of multiple compressed dictionaries in the dictionary matrix, the following will be described separately:

[0061] As an implementation, the compressed dictionary includes row vectors and column vectors. Multiple compressed dictionaries are stored in multiple different column vectors in the dictionary matrix. The number of hash kernels is the same as the number of elements in the column vector. Each hash kernel queries multiple column vectors of the compressed dictionary in parallel to achieve parallel query and improve the query efficiency. As an example, Figure 3 is one of the schematic diagrams of a dictionary matrix provided by an embodiment of the present application. As Figure 3 shown, in order to save the occupied storage resources and improve the compression performance, the size of the dictionary matrix is [(X + 1), (X + 1)], that is, the dictionary matrix includes (X + 1) row vectors and (X + 1) column vectors. (X + 1) compressed dictionaries are stored in (X + 1) different column vectors in the dictionary matrix, and the number of hash kernels is also (X + 1). The (X + 1) hash kernels are respectively named: hash kernel 0 (core0), hash kernel 1 (core1), hash kernel 2 (core2), and sequentially numbered until hash kernel X (coreX); the (X + 1) compressed dictionaries are respectively named: compressed dictionary 0, compressed dictionary 1, and sequentially numbered until compressed dictionary X. Among them, the data stored in multiple compressed dictionaries is the same. Among them, hash kernel 0 corresponds to the 0th column vector in the dictionary array, hash kernel 1 corresponds to the first column vector in the dictionary array, and so on. Hash kernel X corresponds to the Xth column vector in the dictionary array. Taking compressed dictionary 1 as an example, RAM00, RAM10 until RAMX0 in the dictionary matrix constitute the first column vector in the dictionary matrix. That is to say, all the data in compressed dictionary 1 is stored in RAM00, RAM10 until RAMX0. Thus, when querying the corresponding compressed dictionary according to the hash values of the first data corresponding to multiple hash kernels, multiple hash kernels can use the hash values of multiple first data to query the data of multiple column vectors of the dictionary in parallel to determine the hit results of the hash values of multiple first data in the compressed dictionary, improving the query efficiency.

[0062] As another implementation, as an example, Figure 4This is the second schematic diagram of a dictionary matrix provided by an embodiment of the present application. As Figure 4 shown, the size of the dictionary matrix is [(X + 1), (X + 1)], that is, the dictionary matrix includes (X + 1) row vectors and (X + 1) column vectors. (X + 1) compressed dictionaries are stored in (X + 1) different row vectors in the dictionary matrix, and the number of hash kernels is also (X + 1). The same multiple compressed dictionaries are stored in multiple row vectors. For specific reference, please refer to Figure 4 and the relevant explanations for Figure 3 above. The principle is the same and will not be elaborated here. It can realize synchronous query of the data in all row vectors of the dictionary for the hash values of multiple first data to determine the hit results of the multiple first data in the compressed dictionary, improving the query efficiency.

[0063] Step 203: Parallelly query the multiple compressed dictionaries stored in the dictionary matrix according to the hash values of the multiple first data, and determine the position information associated with the hash value of each first data.

[0064] In the embodiment of the present application, each first data has a corresponding compressed dictionary. Through parallel query, the query efficiency can be improved. For each first data, query the corresponding compressed dictionary in the dictionary matrix to determine the hash value that matches the hash value of the first data from the data stored in the compressed dictionary, and use the position information corresponding to the matching hash value as the position information associated with the hash value of the first data. Among them, the position information indicates the position information for reading data from the data to be compressed.

[0065] Step 204: Use the position information associated with the hash value of each first data as the matching data of the hash value of each first data in the corresponding compressed dictionary.

[0066] In the embodiment of the present application, for the hash value of each first data, use the position information associated with the hash value of the first data as the matching data of the hash value of the first data in the corresponding compressed dictionary.

[0067] Step 205: Determine the corresponding compression sequence of each first data according to the matching data of the hash value of the multiple first data in the corresponding compressed dictionary.

[0068] In an embodiment of the present application, according to the position information associated with the hash value of each first data, a second data is read from the data to be compressed, where the number of bytes included in the second data is the same as that of the first data. According to the comparison results of each first data and the corresponding second data, a compression sequence corresponding to each first data is determined. As an implementation manner, in response to at least one first target data among a plurality of first data being the same as the corresponding second data, it is determined that the data hits. According to each first target data and the number of times the current data misses for the first time, a compression sequence corresponding to each first target data is determined, each second target data is encoded, and the encoding result of each second target data is used as the compression sequence corresponding to each second target data, where the second target data is the data among the plurality of first data other than at least one first target data. Among them, the comparison result of each first data and the corresponding second data indicates whether each first data is the data that first appears in the corresponding dictionary. According to whether each first data is the data that first appears in the corresponding dictionary, it is determined whether the plurality of first data corresponding to the plurality of hash kernels hit the data in the compression dictionary. According to whether the data hits, it is determined whether each first data is to be compressed or encoded to obtain the corresponding compression sequence, so as to implement the compression processing of the data to be compressed.

[0069] It should be noted that in the related art, there is one compression dictionary and serial access to the compression dictionary is required. In the present application, multiple hash kernels correspond to multiple compression dictionaries, and parallel access can be achieved. For processing the same amount of compressed data, the compression time used by the compression method of the present application is greatly reduced, and the compression efficiency is improved.

[0070] In the compression method of the embodiment of the present application, in the scenario of data compression based on a compression dictionary, multiple compression dictionaries are set for multiple hash kernels, and the hash values of the first data corresponding to the multiple hash kernels are queried in parallel for the corresponding multiple compression dictionaries, reducing the conflict problem of multiple hash kernels querying a single compression dictionary. At the same time, querying the corresponding compression dictionaries for multiple hash values in parallel realizes synchronous processing of multiple first data, improving the efficiency of data compression.

[0071] Based on the above embodiment, Figure 5A is a schematic flowchart of another compression method provided by the embodiment of the present application, as Figure 5A shown, the method includes the following steps:

[0072] Step 501, determine the hash values obtained by each hash kernel in the compression engine for hashing the first data to be processed.

[0073] Step 502, query the corresponding compression dictionaries in parallel according to the hash values of the multiple first data, and determine the matching data of the hash values of each first data in the corresponding compression dictionaries.

[0074] Among them, steps 501 to 502 can refer to the relevant explanations in the foregoing embodiments. The principles are the same and will not be elaborated here.

[0075] Step 503: Read the second data from the data to be compressed according to the position information associated with the hash value of each first data.

[0076] Among them, the second data and the first data include the same number of bytes.

[0077] In the embodiments of the present application, for the position information associated with the hash value of each first data, the second data with the same number of bytes as the first data is read from the data to be compressed according to the position information.

[0078] Step 504: Compare each first data with the corresponding second data. In response to at least one first target data being the same as the corresponding second data, it is determined that the data hits.

[0079] Among them, the data hit indicates that at least one first target data are all the data that repeatedly appear in the data to be compressed.

[0080] In the embodiments of the present application, for each first data, the first data is compared with the corresponding second data of the first data. If at least one first target data among the multiple first data is the same as the corresponding second data, it means that at least one first target data repeatedly appears in the data to be compressed. That is to say, for each first target data, the information of the historical data that is the same as the first data has been stored in the compression dictionary. The information includes the hash value of the data and the storage position in the data to be compressed. It can also be considered that each first target data repeatedly appears in the compression dictionary. Therefore, for the found duplicate data, it is replaced with an index pointing to the data, so as to achieve the compression effect. In the actual scenario, in order to improve the compression efficiency, based on the first target data, the longest data to be compressed corresponding to the first target data is determined from the data to be compressed for compression.

[0081] Step 505: In response to the data hitting and the first number of times that the current data does not hit being greater than or equal to the number threshold, for each first target data, search in the first direction and the second direction respectively based on the position of the first target data in the data to be compressed to obtain the first candidate data to be matched.

[0082] Step 506: Search in the first direction and the second direction respectively based on the position of the second data corresponding to the first target data in the data to be compressed to obtain the second candidate data to be matched.

[0083] Step 507: Determine the third target data to be compressed corresponding to the first target data according to the first candidate data and the second candidate data, and compress the third target data to obtain a compression sequence corresponding to the first target data.

[0084] In the embodiment of the present application, for each first target data, it is necessary to traverse and determine the longest data to be compressed from the data to be compressed, and use a set dictionary compression algorithm to compress the longest data to be compressed to obtain a compression sequence corresponding to the first target data. Among them, in the case of data hit, obtain the first number of times that the current data is not hit. If the first number is greater than or equal to the number threshold, it means that the current number of times of non-hit is relatively large, and there is a situation where there are consecutive data non-hits in the front and the first number is greater than the number threshold. That is to say, the position offset of data reading was increased in the previous time, and increasing the position offset of data reading can improve the compression efficiency. However, in the related art, the longest compressed data is determined by backward search. Therefore, increasing the position offset of data reading will cause some data corresponding to the position offset not to be considered for compression, resulting in a decrease in the compression ratio. In order to reduce the loss of the compression ratio, in the embodiment of the present application, forward search is added on the basis of backward search to determine the longest compressible length, so as to ensure the compression ratio while improving the compression rate.

[0085] As an implementation, update the value of the first count to 0, and use the first target data as the first initial data to perform the traversal process of the query. The traversal process includes: based on the position of the first initial data, search for the first candidate data to be matched in the first direction and the second direction from the data to be compressed. The first candidate data includes the first initial data, the first set-length data adjacent to the first initial data in the first reverse direction, and the second set-length data adjacent to the first initial data in the second direction. According to the position of the second data corresponding to the first target data in the data to be compressed, search for the second candidate data to be matched in the first direction and the second direction from the data to be compressed. The second candidate data includes the second data, the first set-length data adjacent to the second data in the first reverse direction, and the second set-length data adjacent to the second data in the second direction. Compare the first candidate data and the second candidate data. In response to the first candidate data and the second candidate data being the same, update the first candidate data to the first initial data, and repeat the traversal process until the first candidate data and the second candidate data are different. Use the first candidate data obtained in the previous execution of the traversal process of the last time as the third target data to be compressed corresponding to the first target data, and compress the third target data to be compressed corresponding to the first target data to obtain the corresponding compressed sequence. In the embodiments of the present application, when determining each first target data, in order to improve the compression efficiency and reduce the storage of redundant data, it is necessary to find the longest data sequence that matches based on the first target data in the data to be compressed. In the related art, it is to query in the second direction, that is, backward (to the right) based on the position of the first target data in the data to be compressed. However, in the present application, since the position offset of data reading is increased, some data is not processed. Therefore, in order to improve the compression ratio, it is necessary to query in the first direction and the second direction respectively based on the position of the first target data in the data to be compressed to determine the data with the longest matching data length, that is, the third target data, which meets the requirement of the compression ratio while improving the compression speed.

[0086] As an example, as Figure 5B shown, the first direction and the second direction are opposite.

[0087] Step 508, in response to the data hitting and the first count of the currently unhit data being less than the count threshold, for each first target data, search in the second direction based on the position of the first target data in the data to be compressed to obtain the third candidate data to be matched.

[0088] Step 509, query in the second direction based on the position of the second data corresponding to the first target data in the data to be compressed to obtain the fourth candidate data to be matched.

[0089] Step 510: Determine the fourth target data to be compressed corresponding to the first target data according to the third candidate data and the fourth candidate data, and compress the fourth target data to obtain the compression sequence corresponding to the first target data.

[0090] In the embodiment of the present application, in the case of data hit, obtain the first number of times that the current data misses. If the first number is less than the number threshold, it means that the number of current misses is small, and the position offset of multiple hash kernels for data reading does not increase. Then update the value of the first number to 0, and for each first target data, search in the data to be compressed in the second direction based on the position of the first target data in the data to be compressed to obtain the third candidate data to be matched. Search in the data to be compressed in the second direction based on the position of the second data corresponding to the first target data in the data to be compressed to obtain the fourth candidate data to be matched. Determine the fourth target data to be compressed corresponding to the first target data according to the matching results of the third candidate data and the fourth candidate data. Among them, in the process of matching through query, the traversal process in the foregoing steps is also executed, and the principle is similar, so it will not be elaborated in this embodiment. Query in the second direction based on the position of the first target data in the data to be compressed to determine the fourth target data to be compressed. The fourth target data is the sequence of the data with the largest length obtained by query matching. Compress the fourth target data to be compressed to obtain the compression sequence corresponding to the first target data.

[0091] Step 511: Encode each second target data, and use the encoding results of each second target data as the compression sequences corresponding to each second target data.

[0092] In the embodiment of the present application, for each second target data, since each second target data is not the data that repeatedly appears in the compression dictionary, that is, the information of the historical data identical to the second target data is not stored in the compression dictionary. The information includes the hash value of the data and the storage position in the data to be compressed. It can also be considered that each second target data does not repeatedly appear in the compression dictionary. Therefore, it is impossible to use an index or encoding to replace the second target data, so a set encoding method is used for encoding to obtain the corresponding compression sequence.

[0093] In the compression method of the embodiment of the present application, in the scenario of data compression based on a compression dictionary, multiple compression dictionaries are set for multiple hash kernels, and the hash values of the first data corresponding to the multiple hash kernels are queried in parallel for the corresponding multiple compression dictionaries, reducing the conflict problem of a single compression dictionary query by multiple hash kernels. At the same time, the multiple hash values are queried in parallel for the corresponding compression dictionaries to synchronously process multiple first data, improving the data compression efficiency and compression performance. At the same time, during the compression process, by increasing forward matching, while improving the compression rate, the loss of the compression ratio is reduced, the compression effect is improved, and the storage space occupied by the data is reduced.

[0094] Based on the above embodiment, Figure 6 is a schematic flowchart of another compression method provided by the embodiment of the present application. As Figure 6 shown, the method includes the following steps:

[0095] Step 601, determine the hash values obtained by performing hash processing on the first data to be processed by each hash kernel in the compression engine.

[0096] Step 602, query the corresponding compression dictionaries in parallel according to the hash values of the multiple first data, and determine the matching data of the hash values of each first data in the corresponding compression dictionary.

[0097] Step 603, read the second data from the data to be compressed according to the position information associated with the hash value of each first data.

[0098] Wherein, the number of bytes included in the second data is the same as that of the first data.

[0099] Among them, steps 601 to 603 can refer to the relevant explanations in the foregoing embodiment, with the same principle, which will not be elaborated here.

[0100] Step 604, compare each first data with the corresponding second data, and in response to all the multiple first data being different from the corresponding second data, determine that the data is not hit.

[0101] In the embodiment of the present application, when each first data is compared with the corresponding second data, if all the multiple first data are different from the corresponding second data, it is determined that the data is not hit. That is to say, the multiple first data are all the data that first appear in the data to be compressed, that is, the information of historical data identical to each first data is not stored in the compression dictionary, so it is considered that each first data does not appear repeatedly in the compression dictionary.

[0102] Step 605, encode each first data, and use the encoding result of each first data as the compression sequence corresponding to each first data.

[0103] Thus, for each first data, since the first data does not appear repeatedly in the compression dictionary, it is impossible to use an index or encoding to replace the corresponding first data. Therefore, after encoding using the set encoding method, the compression sequence corresponding to the first data is obtained.

[0104] Step 606: Obtain the current unhit first count, and update the first count according to the number of hash kernels to obtain the second count.

[0105] In the embodiment of the present application, if the data is not hit, obtain the current stored unhit first count, and update the first count according to the number of hash kernels to obtain the second count. As an example, the first count is miss_cnt1, and the number of hash kernels is 4. Then the calculation formula for the second count miss_cnt2 is as follows:

[0106] miss_cnt2 = miss_cnt1 + 4.

[0107] Step 607: Compare the second count with the count threshold. In response to the second count being greater than or equal to the count threshold, increase the position offset of data reading from the first position offset to the second position offset.

[0108] Among them, the position offset of data reading indicates the position offset corresponding to the first data to be processed read from the data to be compressed by multiple hash kernels twice.

[0109] Among them, the count threshold is set based on experience, and the specific size of the count threshold is not limited in this embodiment.

[0110] In the embodiment of the present application, compare the second count with the count threshold. If the second count is greater than or equal to the count threshold, it means that the current first data and the corresponding second data are continuously mismatched more frequently. Therefore, it is necessary to increase the granularity of the compression step, that is, it is necessary to increase the position offset of reading data from the data to be compressed from the first position offset to the second position offset.

[0111] As an example, this time is the Nth time, the previous time is the (N - 1)th time, and the time before the previous time is the (N - 2)th time. For example, Figure 7As shown, the data is divided into byte units. The offset of the first position of the data read from the data to be compressed in the (N - 1)th time relative to the (N - 2)th time is 2 bytes. By increasing the offset of the data reading position, the starting position of the data read from the data to be compressed in the Nth time is offset by 2 bytes relative to the ending position of the data read from the data to be compressed in the (N - 1)th time, that is, the second position offset is 4 bytes. In the case of data hit, if the number of hits is large, it indicates that the current position offset may be small. To improve the compression rate, by increasing the granularity of the position offset, the data to be compressed can be processed quickly. As an example, each time multiple hash kernels obtain data from the data to be compressed, there is a starting position. Then, based on this starting position, the multiple hash kernels are offset and data is read, so that each time multiple hash kernels obtain the corresponding data to be processed. As an example, in combination with Figure 7 it is described. Taking 2 hash kernels as an example, each time the 2 hash kernels read a total of 3 bytes of data from the data to be compressed. Each time the 2 hash kernels obtain data from the data to be compressed, there is an interval of one data reading between them. The offset of data reading between the hash kernels is a fixed value and does not change during the compression process.

[0112] Step 608: Determine the second starting position of the data read from the data to be compressed next time according to the second position offset and the position of the data that has been read in the data to be compressed currently.

[0113] As an example, as Figure 7 shown, the position indicated by A is the position of the data that has been read in the data to be compressed currently. If the second position offset is 4, then the second starting position is determined to be the position indicated by B.

[0114] Step 609: In response to the second number being less than the number threshold, maintain the position offset of the data reading as the first position offset.

[0115] In the embodiment of the present application, if the second number is less than the number threshold, it indicates that the data hit situation meets the requirements. Then, maintain the position offset of the data reading as the first position offset. Thus, the second starting position of the data read from the data to be compressed next time is miss_cnt2 = miss_cnt1 + 2.

[0116] It should be noted that in the case of determining the second starting position of the data read from the data to be compressed next time, that is, obtaining the new data to be processed by each hash kernel from the second starting position, the processing method of the new data can refer to the relevant explanations of the first data processing in the foregoing embodiments. The principle is the same and will not be elaborated in this embodiment until all the data in the data to be compressed is processed and the entire compression sequence is output.

[0117] It should be noted that steps 606 to 609 can be executed before step 605, after step 605, or in parallel. There is no limitation in this embodiment.

[0118] In the compression method of the embodiment of the present application, the number of data hits is monitored. When the number of data hits is greater than the set threshold, it indicates that there is a current situation of continuous mismatches. The position offset of data reading can be adjusted to increase the interval for multiple hash kernels to read data from the data to be compressed next time, so as to accelerate the compression speed.

[0119] Based on the above embodiment, in response to a data miss, it is necessary to update multiple compression dictionaries in the dictionary matrix through each hash kernel based on the hash value of the corresponding first data and the position information of each first data in the data to be compressed. As an implementation manner, in response to a data miss, the target vectors in the dictionary matrix to be updated by each hash kernel are determined, and the hash value of each first data and the position information of each first data in the data to be compressed are used to synchronously update the target vectors corresponding to each hash kernel, where the target vector is a target row vector or a target column vector. For different storage methods of the compression dictionary, the update method is described as follows:

[0120] As a first implementation manner, as an example, as Figure 3 shown, multiple compression dictionaries are stored in multiple different column vectors in the dictionary matrix. (X + 1) hash kernels correspond to (X + 1) compression dictionaries. The number of (X + 1) elements included in each compression dictionary is equal to the number of hash kernels. That is to say, the number of rows in the dictionary matrix is the same as the number of hash kernels. Thus, each hash kernel can synchronously update the target row data in multiple compression dictionaries, realizing the synchronous update of multiple compression dictionaries. In response to a data miss, the target row vectors to be updated by each hash kernel in the dictionary matrix are determined, and the data stored in the target row vectors corresponding to each hash kernel are synchronously updated by using the hash value of each first data and the position information of each first data in the data to be compressed. Through the synchronous update of each hash kernel, the update speed is improved, and it is ensured that multiple compression dictionaries are updated synchronously.

[0121] As a second implementation manner, as Figure 4As shown, multiple compression dictionaries are stored in multiple different row vectors in a dictionary matrix. (X + 1) hash kernels correspond to (X + 1) compression dictionaries. In response to a data miss, the target column vectors to be updated for each hash kernel in the dictionary matrix are determined. Using the hash values of each first data and the position information of each first data in the data to be compressed, the data stored in the target column vectors corresponding to each hash kernel is synchronously updated. The data stored in one target column vector is from multiple compression dictionaries, realizing the synchronous update of multiple compression dictionaries. Through the synchronous update of each hash kernel, the update speed is improved, and it is ensured that multiple compression dictionaries are updated synchronously.

[0122] Based on the above embodiments, an embodiment of the present application also proposes a compression method. As an implementation, before determining the hash values obtained by each hash kernel in the compression engine for hashing the first data to be processed, it further includes:

[0123] Obtain the target command to be executed from at least one compression command stored in the command queue. Among them, the at least one compression command is generated by the processor and stored in the command queue. Specifically, the processor generates at least one compression command and stores the at least one compression command in the command queue. Obtain the target command to be executed from the command queue, and according to the storage location of the data to be compressed carried in the target command, obtain the data to be compressed. Determine the first data to be processed by each hash kernel in the compression engine from the data to be compressed. During compression, as long as the command queue of the compression module is not full, the processor can continuously fill the subsequent data to be compressed into the command queue without waiting for the current data to be compressed to be executed, realizing support for multi-task processing, improving the interaction performance of the software, and thus improving the compression efficiency.

[0124] Based on the foregoing embodiments, an embodiment of the present application provides a compression system. Figure 8 For the structural schematic diagram of a compression system provided by an embodiment of the present application, as Figure 8 shown, the compression system 100 includes a compression module 110.

[0125] The compression module 110 is used to determine the hash values obtained by each hash kernel in the compression engine for hashing the first data to be processed, parallelly query the corresponding compression dictionaries according to the hash values of multiple first data, determine the matching data of the hash values of each first data in the corresponding compression dictionaries, and determine the compression sequences corresponding to each first data according to the matching data of the hash values of multiple first data in the corresponding compression dictionaries.

[0126] Among them, the relevant explanations in the foregoing embodiments also apply to this embodiment. The principles are the same and will not be elaborated here.

[0127] Based on the above embodiments, as an implementation, the compression system 100 includes a processor 120 connected to a compression module 110.

[0128] The processor 120 is configured to generate at least one compression command and store the at least one compression command in a command queue in the compression module 110.

[0129] The compression module 110 is configured to obtain a target command to be executed from at least one compression command stored in the command queue, obtain the data to be compressed according to the storage location of the data to be compressed carried in the target command, and determine first data to be processed by each hash core from the data to be compressed.

[0130] As an example, Figure 9 FIG. is a schematic structural diagram of another compression system provided by an embodiment of the present application. As shown in Figure 9 FIG., the compression system 100 includes: a compression module 110, a processor 120, a bus 130, and a storage unit 140. The compression module 110 includes a command queue 113 and a compression engine. Among them, the processor 120 is, for example, a CPU, and the storage unit 140 is, for example, a DDR. The processor 120 collects information on the data to be compressed in the storage unit 140 and queries whether the compression commands in the command queue 113 are stored full. When the command queue 113 is not full and there is data to be compressed in the storage unit 140, the processor 120 generates multiple compression commands and stores the multiple compression commands in the command queue 113 through the bus 130. Further, when the compression engine 111 is idle, the compression engine 111 obtains a target command to be executed from the command queue 113, obtains the data to be compressed according to the storage location of the data to be compressed carried in the target command, and determines first data to be processed by each hash core in the compression engine 111 from the data to be compressed. For the processing process of the multiple first data, reference may be made to the relevant explanations and descriptions of the beneficial effects in the foregoing embodiments. The principle is the same and will not be elaborated here.

[0131] It should be noted that during compression, as long as the command queue 113 of the compression module 110 is not full, the processor 120 can continuously fill the subsequent data to be compressed into the command queue 113 without waiting for the current data to be compressed to be executed, so as to support multi-task processing, improve the interaction performance of the software, and thus improve the compression efficiency.

[0132] Among them, the relevant explanations in the foregoing embodiments also apply to this embodiment. The principle is the same and will not be elaborated here.

[0133] Based on the above embodiments, as an implementation, the compression module is further configured to:

[0134] Parallelly query multiple compressed dictionaries stored in a dictionary matrix according to the hash values of multiple pieces of the first data, and determine the position information associated with the hash value of each piece of the first data;

[0135] Use the position information associated with the hash value of each piece of the first data as the matching data of the hash value of each piece of the first data in the corresponding compressed dictionary.

[0136] Based on the above embodiment, as an implementation manner, the compression module is further configured to:

[0137] According to the corresponding relationship between each hash kernel and the compressed dictionaries stored in the dictionary matrix, determine the compressed dictionary corresponding to each hash kernel from the dictionary matrix, where multiple compressed dictionaries are stored in multiple different vectors in the dictionary matrix, and the vector is a row vector or a column vector.

[0138] Based on the above embodiment, as an implementation manner, the compression module is further configured to:

[0139] Read a second data from the data to be compressed according to the position information associated with the hash value of each piece of the first data; where the number of bytes included in the second data is the same as that of the first data; and each piece of the first data is read from the data to be compressed;

[0140] Determine the compression sequence corresponding to each piece of the first data according to the comparison results of multiple pieces of the first data and the corresponding second data.

[0141] Based on the above embodiment, as an implementation manner, the compression module is further configured to:

[0142] Compare each piece of the first data with the corresponding second data;

[0143] In response to at least one piece of first target data being the same as the corresponding second data, determine that the data is hit;

[0144] Determine the compression sequence corresponding to each piece of the first target data according to each piece of the first target data;

[0145] Encode each piece of the second target data, and use the encoding result of each piece of the second target data as the compression sequence corresponding to each piece of the second target data; where the second target data is the data among the multiple pieces of the first data other than the at least one piece of first target data.

[0146] Based on the above embodiment, as an implementation manner, the compression module is further configured to:

[0147] In response to data hit, and when the first number of current data misses is greater than or equal to the number threshold, for each first target data, in the data to be compressed, based on the position of the first target data in the data to be compressed, search in the first direction and the second direction respectively to obtain first candidate data to be matched; wherein, the first direction and the second direction are opposite;

[0148] In the data to be compressed, based on the position of the second data corresponding to the first target data in the data to be compressed, query in the first direction and the second direction respectively to obtain second candidate data to be matched;

[0149] According to the first candidate data and the second candidate data, determine third target data to be compressed corresponding to the first target data;

[0150] Compress the third target data to be compressed to obtain a compression sequence corresponding to the first target data.

[0151] Based on the above embodiments, as an implementation manner, the compression module is further configured to:

[0152] In response to data hit, and when the first number of current data misses is less than the number threshold, for each first target data, in the data to be compressed, based on the position of the first target data in the data to be compressed, search in the second direction to obtain third candidate data to be matched;

[0153] In the data to be compressed, based on the position of the second data corresponding to the first target data in the data to be compressed, search in the second direction to obtain fourth candidate data to be matched;

[0154] According to the third candidate data and the fourth candidate data, determine fourth target data to be compressed corresponding to the first target data;

[0155] Compress the fourth target data to be compressed to obtain a compression sequence corresponding to the first target data.

[0156] Based on the above embodiments, as an implementation manner, the compression module is further configured to:

[0157] Compare each first data with the corresponding second data;

[0158] In response to all first data being different from the corresponding second data, determine data miss; encode each of the first data, and use the encoding results of each of the first data as the compression sequences corresponding to each of the first data.

[0159] Based on the above embodiments, as an implementation, the compression module is further configured to:

[0160] In response to a data miss, determine the target vectors to be updated for each hash core in the dictionary matrix;

[0161] Synchronously update the target vectors corresponding to each hash core by using the hash values of each of the first data and the position information of each of the first data in the data to be compressed.

[0162] Based on the above embodiments, as an implementation, the compression module is further configured to:

[0163] In response to a data miss, obtain the first number of times of the current data miss;

[0164] Update the first number of times according to the number of hash cores to obtain a second number of times;

[0165] Compare the second number of times with a number threshold;

[0166] In response to the second number of times being greater than or equal to the number threshold, increase the position offset of data reading from the first position offset to the second position offset; determine the second starting position for reading data from the data to be compressed next according to the second position offset and the end position of reading data from the current multiple hash cores from the data to be compressed.

[0167] Based on the above embodiments, as an implementation, the compression module is further configured to:

[0168] In response to the second number of times being less than the number threshold, maintain the position offset of data reading as the first position offset.

[0169] The relevant explanations and beneficial effects in the foregoing embodiments are also applicable to this embodiment, and the principles are the same, so they will not be elaborated here.

[0170] To implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the foregoing method embodiment is implemented.

[0171] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0172] Refer to Figure 10, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0173] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0174] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0175] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0176] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of a touch or swipe action but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0177] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0178] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0179] The sensor component 814 includes one or more sensors for providing a status assessment of various aspects of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the electronic device 800, the relative positioning of components, such as the display and the keypad of the electronic device 800. The sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. The sensor component 814 can include a proximity sensor that is configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0180] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a communication standard-based wireless network, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0181] In an exemplary embodiment, the electronic device 800 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by a processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0183] To implement the above embodiments, the present application also proposes a chip, including: The chip includes a processing circuit configured to execute the method provided in the foregoing embodiments.

[0184] Figure 11 It is a schematic structural diagram of a chip proposed in an embodiment of the present application. Reference can be made to Figure 11 the schematic structural diagram of the chip 1100 shown, but not limited thereto.

[0185] The chip 1100 includes a processing circuit 1101 configured to execute any of the above methods.

[0186] In some embodiments, the chip 1100 further includes one or more interface circuits 1102. Optionally, the interface circuit 1102 is connected to the memory 1103. The interface circuit 1102 can be used to receive signals from the memory 1103 or other devices, and the interface circuit 1102 can be used to send signals to the memory 1103 or other devices. For example, the interface circuit 1102 can read the instructions stored in the memory 1103 and send the instructions to the processing circuit 1101.

[0187] In some embodiments, the interface circuit 1102 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 1101 performs other steps.

[0188] In some embodiments, terms such as interface circuit, interface, transceiver pin, transceiver, etc. may be used interchangeably.

[0189] In some embodiments, the chip 1100 further includes one or more memories 1103 for storing instructions. Optionally, all or part of the memories 1103 may be outside the chip 1100.

[0190] To implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the foregoing method embodiments is implemented.

[0191] To implement the above embodiments, the present application also proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the method described in the foregoing method embodiments is implemented.

[0192] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0193] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0194] Any process or method description represented in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process. The scope of the preferred embodiments of the present application includes additional implementations where functions may be executed not in the order shown or discussed, including in a substantially simultaneous manner according to the functions involved or in a reverse order, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0195] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing a logical function and can be embodied specifically in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. As used in this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0196] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following techniques known in the art or a combination thereof can be used: discrete logic circuits having logic gates for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0197] Those of ordinary skill in the art can understand that all or part of the steps carried out in implementing the above-described embodiment methods can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0198] In addition, in each of the embodiments of the present application, the functional units can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0199] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A compression method, characterized in that, Including: Determining hash values obtained by performing hash processing on first data to be processed by respective hash kernels in a compression engine; Parallelly querying corresponding compression dictionaries according to the hash values of multiple first data, and determining matching data of the hash values of respective first data in the corresponding compression dictionaries; Determining compression sequences corresponding to respective first data according to the matching data of the hash values of the multiple first data in the corresponding compression dictionaries.

2. The method according to claim 1, wherein The parallelly querying corresponding compression dictionaries according to the hash values of multiple first data, and determining matching data of the hash values of respective first data in the corresponding compression dictionaries includes: Parallelly querying multiple compression dictionaries stored in a dictionary matrix according to the hash values of multiple said first data, and determining position information associated with the hash values of respective said first data; Taking the position information associated with the hash value of each first data as the matching data of the hash value of each first data in the corresponding compression dictionary.

3. The method according to claim 2, wherein Before the parallelly querying corresponding multiple compression dictionaries in the dictionary matrix according to the hash values of multiple said first data, it includes: Determining compression dictionaries corresponding to respective hash kernels from the dictionary matrix according to the corresponding relationship between each hash kernel and the compression dictionaries stored in the dictionary matrix; wherein, multiple compression dictionaries are stored in multiple different vectors in the dictionary matrix, and the vector is a row vector or a column vector.

4. The method according to claim 3, wherein The determining compression sequences corresponding to respective first data according to the matching data of the hash values of the multiple first data in the corresponding compression dictionaries includes: Reading second data from the data to be compressed according to the position information associated with the hash value of each first data; wherein, the second data and the first data have the same data length; and wherein, each of the first data is read from the data to be compressed; Determining compression sequences corresponding to respective first data according to comparison results of multiple said first data and corresponding second data.

5. The method according to claim 4, characterized in that The determining compression sequences corresponding to respective first data according to comparison results of multiple said first data and corresponding second data includes: Comparing each of the first data with the corresponding second data; In response to at least one first target data among the multiple first data being the same as the corresponding second data, determining that data is hit; Determining compression sequences corresponding to respective first target data according to each first target data and the first number of times that current data is not hit; Encoding each second target data, and taking the encoding result of each second target data as the compression sequence corresponding to each second target data; wherein, the second target data is data other than the at least one first target data among the multiple first data.

6. The method according to claim 5, characterized in that, The determining compression sequences corresponding to respective first target data according to each first target data and the first number of times that current data is not hit includes: In response to a data hit and when the first number of current data misses is greater than or equal to the number threshold, for each first target data, search in the to-be-compressed data in the first direction and the second direction respectively based on the position of the first target data in the to-be-compressed data to obtain first candidate data to be matched; wherein, the first direction and the second direction are opposite; In the to-be-compressed data, query in the first direction and the second direction respectively based on the position of the second data corresponding to the first target data in the to-be-compressed data to obtain second candidate data to be matched; Determine third target data to be compressed corresponding to the first target data according to the first candidate data and the second candidate data; Compress the third target data to be compressed to obtain a compression sequence corresponding to the first target data.

7. The method according to claim 5, characterized in that, The determining the compression sequences corresponding to the respective first target data according to each of the first target data and the first number of current data misses includes: In response to a data hit and when the first number of current data misses is less than the number threshold, for each first target data, search in the second direction in the to-be-compressed data based on the position of the first target data in the to-be-compressed data to obtain third candidate data to be matched; In the to-be-compressed data, search in the second direction based on the position of the second data corresponding to the first target data in the to-be-compressed data to obtain fourth candidate data to be matched; Determine fourth target data to be compressed corresponding to the first target data according to the third candidate data and the fourth candidate data; Compress the fourth target data to be compressed to obtain a compression sequence corresponding to the first target data.

8. The method according to claim 4, wherein The determining the compression sequences corresponding to the respective first data according to the comparison results of the multiple first data and the corresponding second data includes: Compare each first data with the corresponding second data; In response to all the multiple first data being different from the corresponding second data, determine a data miss; Encode each of the first data, and use the encoding results of each of the first data as the compression sequences corresponding to each of the first data.

9. The method according to claim 8, wherein The method further includes: In response to a data miss, determine target vectors in the dictionary matrix to be updated for each hash kernel; Synchronously update the target vectors corresponding to each hash kernel by using the hash values of each of the first data and the position information of each of the first data in the to-be-compressed data.

10. The method according to claim 8, characterized in that, The method further includes: In response to a data miss, obtain the first number of current data misses; Update the first number to obtain a second number according to the number of hash kernels; Compare the second number with the number threshold; In response to the second number being greater than or equal to the number threshold, increase the position offset of data reading from the first position offset to the second position offset; Determine a second starting position for the next multiple hash kernels to read data from the data to be compressed according to the second position offset and the end position of the current multiple hash kernels reading data from the data to be compressed.

11. The method according to claim 10, wherein The method further includes: In response to the second number being less than the number threshold, maintain the position offset of data reading as the first position offset.

12. The method according to any one of claims 1 to 11, characterized in that, Before determining the hash values obtained by performing hash processing on the first data to be processed by each hash kernel in the compression engine, it further includes: Obtain a target command to be executed from at least one compression command stored in the command queue; wherein, the at least one compression command is generated by the processor and stored in the command queue. Obtain the data to be compressed according to the storage position of the data to be compressed carried in the target command. Determine the first data to be processed by each of the hash kernels from the data to be compressed.

13. A compression system, characterized in that, It includes: A compression module; The compression module is configured to determine the hash values obtained by performing hash processing on the first data to be processed by each hash kernel in the compression engine, parallelly query the corresponding compression dictionary according to the hash values of the multiple first data, determine the matching data of the hash values of each first data in the corresponding compression dictionary, and determine the compression sequence corresponding to each first data according to the matching data of the hash values of the multiple first data in the corresponding compression dictionary.

14. The compression system according to claim 13, wherein The compression system includes a processor connected to the compression module; The processor is configured to generate at least one compression command and store the at least one compression command in the command queue in the compression module. The compression module is configured to obtain a target command to be executed from at least one compression command stored in the command queue, obtain the data to be compressed according to the storage position of the data to be compressed carried in the target command, and determine the first data to be processed by each of the hash kernels from the data to be compressed.

15. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method according to any one of claims 1-12.

16. A chip, characterized in that, The chip includes a processing circuit configured to execute the method according to any one of claims 1-12.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1-12.

18. A computer program product, characterized in that, It includes a computer program which, when executed by the processor, implements the method according to any one of claims 1-12.

Images