A multi-class data intelligent caching method and system
By constructing a paradigm Hoffman tree, using the path encoding of the previous data in the tree instead of traditional path encoding, the compression efficiency of Hoffman encoding is improved, the problem of insufficient buffer storage space is solved, and more efficient data storage is achieved.
Patent Information
- Application Number
- CN202310419100.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-19
- Publication Date
- 2025-08-29
- Estimated Expiration
- 2043-04-19
AI Technical Summary
The storage space of the buffer is limited, and the existing Hoffman encoding compression efficiency is insufficient, which makes it impossible to temporarily store more data in the buffer.
Using the improved Hoffman encoding method, by constructing a paradigm Hoffman tree, the path encoding between the nodes in the tree and the target data is used to replace the traditional path encoding from the root node to the target data, and improve the encoding compression efficiency.
This improves the compression efficiency of Hoffman encoding, reduces the amount of compressed data, and enables more data to be temporarily stored in the buffer.
Smart Images
Figure CN116456117B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data compression, and in particular to a method and system for intelligently caching multiple types of data. Background Art
[0002] A cache is typically a small, high-speed memory device that temporarily stores data within a data processing device. Frequently used data is stored in the cache, allowing the central processing unit (CPU) to directly access this data and improve CPU performance. Because the cache's storage space is limited, compressing the data is necessary to reduce the amount of data needed to store more data.
[0003] Huffman coding is widely used in the field of compression to compress various types of data including images, videos, and audio. Huffman coding performs variable-length encoding on data based on the frequency of the data. Therefore, by improving Huffman coding, the compression efficiency of Huffman coding is improved and the amount of data compressed by Huffman coding is reduced, so that more data can be temporarily stored in the cache. Summary of the Invention
[0004] The present invention provides a multi-type data intelligent caching method and system to solve the existing problems.
[0005] The present invention provides a multi-class data intelligent caching method using the following technical solutions:
[0006] An embodiment of the present invention provides a method for intelligently caching multiple types of data, the method comprising:
[0007] Convert each type of data to be cached in the buffer into a data sequence, obtain all types of data for each type of data, count the frequency of each type of data in the data sequence, sort all types of data in descending order of frequency as a data sequence sequence, and construct a paradigm Huffman tree based on the data sequence sequence and the binary tree;
[0008] Encoding each data in the data sequence in sequence, including: taking the data to be encoded as target data, recording the node corresponding to the target data in the normalized Huffman tree as the target node, and recording the encoding of the shortest path from the root node to the target node as the first encoding of the target data;
[0009] If the target data is the first data in the data sequence, the first code of the target data is used as the target code of the target data; otherwise, the previous data of the target data is recorded as the prefix data:
[0010] In the data sequence, if the prefix data is before the target data, a second code of the target data is obtained according to a path of the prefix data and the target data in the normal form Huffman tree, and a target code of the target data is obtained according to the first code and the second code of the target data; if the prefix data is after the target data, a third code of the target data is obtained according to a path of the prefix data and the target data in the normal form Huffman tree, and a target code of the target data is obtained according to the first code and the third code of the target data;
[0011] A sequence composed of target codes of all data in the data sequence in order is recorded as a coding sequence, and the coding sequence is the compression result of each type of data to be cached.
[0012] Furthermore, the construction of the paradigm Huffman tree includes the following specific steps:
[0013] Construct an N+1-layer binary tree, requiring that the left nodes of all layers in the binary tree and the right nodes of the last layer are leaf nodes, the other nodes in the binary tree are parent nodes, and the node of the first layer in the binary tree is the root node, where N is the number of all kinds of data in the data sequence; the left nodes of all layers except the first two layers and the right nodes of the last layer are recorded as target nodes, and each data in the data sequence is assigned to each target node in turn in order from top to bottom, and the binary tree is assigned codes in the manner of left 1 and right 0, and the obtained binary tree is recorded as a normalized Huffman tree.
[0014] Furthermore, the step of obtaining the second encoding of the target data includes the following specific steps:
[0015] The node corresponding to the prefix data in the normal form Huffman tree is recorded as the start node, the node corresponding to the target data in the normal form Huffman tree is recorded as the end node, and the code of the shortest path from the start node to the end node in the normal form Huffman tree is obtained and recorded as the second code.
[0016] Furthermore, the step of obtaining the target code of the target data according to the first code and the second code of the target data includes the following specific steps:
[0017] If the length of the first code of the target data is less than or equal to the length of the second code of the target data, the first code is used as the target code of the target data; otherwise, the second code is used as the target code of the target data.
[0018] Furthermore, the step of obtaining the third encoding of the target data includes the following specific steps:
[0019] The node corresponding to the target data in the normal form Huffman tree is recorded as the start node, and the node corresponding to the prefix data in the normal form Huffman tree is recorded as the end node. The code of the shortest path from the start node to the end node is obtained in the normal form Huffman tree, and a 1 is added before the code as the third code.
[0020] Furthermore, the step of obtaining the target code of the target data according to the first code and the third code of the target data includes the following specific steps:
[0021] If the length of the first code is less than or equal to the length of the third code, the first code is used as the target code of the target data; otherwise, the third code is used as the target code of the target data.
[0022] An embodiment of the present invention proposes a multi-type data intelligent caching system, including a cache partitioning unit, a data compression unit, and a data storage unit. The cache partitioning unit is used to query the type of data to be cached, set a corresponding identifier for each type of data, and divide the corresponding cache area for each type of data; the data compression unit implements the steps of the above method, and the data storage unit is used to store the compression results of each type of data in the corresponding cache area.
[0023] The beneficial effect of the technical solution of the present invention is that: in conventional Huffman coding, the encoding of the path from the root node to the node corresponding to the target data in the Huffman tree is used as the target coding of the target data. The present invention uses the encoding of the path between the node corresponding to the previous data of the target data in the paradigm Huffman tree and the node corresponding to the target data as the target coding of the target data. Since the node corresponding to the previous data has fewer layers than the root node, in most cases, the encoding of the path between the node corresponding to the previous data and the node corresponding to the target data is shorter than the encoding of the path between the root node and the node corresponding to the target data in conventional Huffman coding, thereby achieving improvement of Huffman coding, improving the compression efficiency of Huffman coding, and reducing the amount of data after compression by Huffman coding, so that more data can be temporarily stored in the buffer. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0025] Figure 1 This is a system block diagram of a multi-type data intelligent caching method of the present invention. DETAILED DESCRIPTION
[0026] To further illustrate the technical means and effectiveness of the present invention in achieving its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a multi-category data intelligent caching method proposed by the present invention. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
[0027] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0028] The specific scheme of the multi-category data intelligent caching method provided by the present invention is described in detail below with reference to the accompanying drawings.
[0029] See also Figure 1 , which illustrates a multi-class data intelligent caching method provided by one embodiment of the present invention. The system includes the following modules:
[0030] S001. Obtain a data sequence and all types of characters, and construct a paradigm Huffman tree based on the frequencies of all types of characters.
[0031] It should be noted that the cache temporarily stores frequently used data, facilitating direct access to this data by the central processing unit (CPU), thereby improving the performance of the CPU. Since the cache has limited storage space, to temporarily store more data in the cache, it is necessary to compress the data to reduce the amount of data, thereby allowing the cache to temporarily store more data. This embodiment improves Huffman coding to increase its compression efficiency and reduce the amount of data compressed by Huffman coding, thereby allowing the cache to temporarily store more data.
[0032] 1. Obtain the data series and all types of data for each type of data.
[0033] The data temporarily stored in the buffer includes multiple types of data including images, videos and audios. Video data is composed of multiple continuous frame images. Therefore, for image and video data, the compression object is the image. The grayscale values of all pixels in the image are arranged in a preset order. The obtained sequence is recorded as a data sequence, and all types of grayscale values in the image are regarded as all types of data.
[0034] In this embodiment, the preset order is serpentine scanning. In other embodiments, the preset order includes but is not limited to Zigzag scanning, Hilbert scanning, raster scanning, and serpentine scanning.
[0035] Audio data is an amplitude time series sequence composed of the amplitude of the sound at each moment. Therefore, the amplitude time series sequence is recorded as a data sequence, and all types of amplitudes in the amplitude time series sequence are regarded as all types of data.
[0036] 2. Construct a normalized Huffman tree based on the frequency of all types of data.
[0037] It should be noted that, in order to ensure the decodability and feasibility of the encoding method of this embodiment, it is necessary to construct a normalized Huffman tree according to the number of all types of data, and it is required that the first two layers of the normalized Huffman tree have no data.
[0038] In this embodiment, the frequency of each type of data in the data sequence is counted, and a sequence consisting of all types of data is sorted in descending order of frequency and recorded as a data sequence sequence.
[0039] Construct an N+1-layer binary tree, requiring that the left nodes of all layers in the binary tree and the right nodes of the last layer are leaf nodes, the other nodes in the binary tree are parent nodes, and the node of the first layer in the binary tree is the root node, where N is the number of all kinds of data in the data sequence; record the left nodes of all layers except the first two layers and the right nodes of the last layer (a total of N nodes) as target nodes, and assign each data in the data sequence to each target node in turn from top to bottom, record "1" as the first digit, and record "0" as the second digit, and assign codes to the binary tree in the manner of left 1 and right 0, that is, the code of the path between each parent node and the left node is the first digit, and the code of the path between each parent node and the right node is the second digit. The obtained binary tree is recorded as a normalized Huffman tree.
[0040] S002. Obtain a coding sequence of the data sequence according to the paradigm Huffman tree.
[0041] It should be noted that, in conventional Huffman coding, the coding for each type of data is obtained based on the Huffman tree, specifically the coding corresponding to the path from the root node to the node corresponding to the data. Therefore, the larger the number of layers of the node corresponding to the data in the Huffman tree, the more nodes the path from the root node to the node corresponding to the data passes through, the longer the code corresponding to the path, and the longer the code of the data. In order to reduce the code length and improve the compression efficiency, the present invention obtains the path between the node corresponding to the previous data and the node corresponding to the next data, and uses the code of the path as the code of the next data. Since the node corresponding to the previous data has a smaller number of layers than the root node, in most cases, the code of the path between the node corresponding to the previous data and the node corresponding to the next data is shorter than the code of the path between the root node and the node corresponding to the next data, thereby improving the compression efficiency.
[0042] According to the paradigm Huffman tree, each data in the data sequence is encoded in sequence, and the data to be encoded is used as the target data, specifically:
[0043] (1) The node corresponding to the target data in the normalized Huffman tree is recorded as the target node, and the code of the shortest path from the root node to the target node is recorded as the first code of the target data;
[0044] It should be noted that the nodes corresponding to the target data are all left nodes in the normalized Huffman tree, and the normalized Huffman tree allocates codes in the manner of left 1 and right 0. Therefore, the first code of the target data is specifically composed of several first digits and one second digit, that is, the last codeword of the first code of the target data is the second digit, and the remaining codewords are all first digits.
[0045] (2) If the target data is the first data in the data sequence, the first code of the target data is used as the target code of the target data; otherwise, the data before the target data is recorded as the prefix data, and the position relationship between the prefix data and the target data in the data sequence is determined:
[0046] In the data sequence, if the prefix data is in front of the target data, the node corresponding to the prefix data in the normal form Huffman tree is recorded as the start node, and the node corresponding to the target data in the normal form Huffman tree is recorded as the end node. The encoding of the shortest path from the start node to the end node is obtained in the normal form Huffman tree and recorded as the second encoding; if the length of the first encoding of the target data is less than or equal to the length of the second encoding of the target data, the first encoding is used as the target encoding of the target data; otherwise, the second encoding is used as the target encoding of the target data.
[0047] It should be noted that the nodes corresponding to the target data and the nodes corresponding to the prefix data are both on the left nodes in the normalized Huffman tree, and the normalized Huffman tree allocates codes in the manner of left 1 and right 0. At the same time, considering the special structure of the normalized Huffman tree (the left nodes of all layers and the right nodes of the last layer are leaf nodes, and the other nodes are parent nodes), the second code is specifically composed of a second digit, several first digits and a second digit, that is, the first codeword and the last codeword of the second code are both second digits, and the remaining codewords are all first digits.
[0048] In the data sequence, if the prefix data is after the target data, the node corresponding to the target data in the normal form Huffman tree is recorded as the start node, and the node corresponding to the prefix data in the normal form Huffman tree is recorded as the end node. The code of the shortest path from the start node to the end node is obtained in the normal form Huffman tree, and a 1 is added before the code as the third code; if the length of the first code is less than or equal to the length of the third code, the first code is used as the target code of the target data; otherwise, the third code is used as the target code of the target data.
[0049] It should be noted that the nodes corresponding to the target data and the nodes corresponding to the prefix data are both on the left nodes in the normalized Huffman tree, and the normalized Huffman tree allocates codes in the manner of left 1 and right 0. At the same time, considering the special structure of the normalized Huffman tree (the left nodes of all layers and the right nodes of the last layer are leaf nodes, and the other nodes are parent nodes), the third code is to add a second digit before the obtained code. Therefore, the third code is specifically composed of two second digits, several first digits and one second digit, that is, the first codeword, the second codeword and the last codeword of the third code are all second digits, and the remaining codewords are all first digits.
[0050] The sequence composed of the target codes of all data in the data sequence in order is recorded as the coding sequence.
[0051] In this embodiment, compared with conventional Huffman coding, in the Huffman tree, the encoding of the path from the root node to the node corresponding to the target data is used as the target coding of the target data. In this embodiment, the encoding of the path between the node corresponding to the previous data of the target data in the paradigm Huffman tree and the node corresponding to the target data is used as the target coding of the target data. Since the node corresponding to the previous data has a smaller number of layers than the root node, in most cases, the encoding of the path between the node corresponding to the previous data and the node corresponding to the target data is shorter than the encoding of the path between the root node and the node corresponding to the target data. By improving the Huffman coding, the compression efficiency of the Huffman coding is improved, and the amount of data compressed by the Huffman coding is reduced, so that more data can be temporarily stored in the buffer.
[0052] S003. Decode the coded sequence to obtain a data sequence.
[0053] It should be noted that the last codeword of the first code is the second digit, and the remaining codewords are all the first digit; the first codeword and the last codeword of the second code are all the second digit, and the remaining codewords are all the first digit; the first codeword, the second codeword and the last codeword of the third code are all the second digit, and the remaining codewords are all the first digit; in summary, the last codeword of the three codes is the second digit, which can be used to determine the end position of the three codes; at the same time, the method for distinguishing the three codes (first code, second code and third code) is: determine the first codeword of the code: if the first codeword is the first digit, the code is the first code; otherwise, continue to determine the second codeword of the code: if the second codeword is the first digit, the code is the second code; if the second codeword is the second digit, the code is the third code.
[0054] In this embodiment, the coded sequence is decoded in order from left to right, specifically:
[0055] 1. Set an empty sequence as the data sequence.
[0056] 2. According to the first codeword and the second codeword in the coding sequence, different types of codes are obtained, specifically:
[0057] If the first codeword is the first digit, the codeword belonging to the second digit closest to the first codeword is obtained as the cutoff codeword, and the sequence composed of all codewords before the cutoff codeword (including the cutoff codeword) is recorded as a code, and the code type is the first code;
[0058] If the first codeword is the second digit and the second codeword is the first digit, then the codeword belonging to the second digit that is closest to the second codeword and follows the second codeword is obtained as the cutoff codeword, and a sequence consisting of all codewords before and including the cutoff codeword is recorded as a code, and the code type is the second code;
[0059] If the first codeword is the second digit and the second codeword is the second digit, then the codeword belonging to the second digit that is closest to the second codeword after the second codeword is obtained as the end codeword, and the sequence composed of all codewords before the end codeword (including the end codeword) is recorded as the code, and the code type is the third code.
[0060] 3. Decode different types of codes according to the paradigm Huffman tree to obtain the target data, specifically:
[0061] If the encoding type is the first encoding, the root node is used as the starting node, the shortest path is obtained according to the encoding, and the data corresponding to the last node connected by the shortest path is used as the target data;
[0062] If the encoding type is the second encoding, the last target data in the data sequence is used as the prefix data, the node corresponding to the prefix data in the normalized Huffman tree is recorded as the starting node, the shortest path is obtained according to the encoding, and the data corresponding to the last node connected by the shortest path is used as the target data;
[0063] If the encoding type is the third encoding, the last target data in the data sequence is used as the prefix data, and the node corresponding to the prefix data in the normal form Huffman tree is recorded as the end node. The shortest path is obtained according to the encoding, and the data corresponding to the first node connected by the shortest path is used as the target data.
[0064] 4. Delete the code from the coding sequence and add the obtained target data to the end of the data sequence.
[0065] 5. Repeat steps 1 to 4 until the encoding sequence is empty. The data sequence obtained at this time is the decoding result of the encoding sequence.
[0066] An embodiment of the present invention proposes a multi-type data intelligent caching system, including a cache partitioning unit, a data compression unit, and a data storage unit. The cache partitioning unit is used to query the type of data to be cached, set a corresponding identifier for each type of data, and divide the corresponding cache area for each type of data; the data compression unit implements the steps of the above method, and the data storage unit is used to store the compression results of each type of data in the corresponding cache area.
[0067] Compared with conventional Huffman coding, the present invention uses the encoding of the path from the root node to the node corresponding to the target data in the Huffman tree as the target coding of the target data, and uses the encoding of the path between the node corresponding to the previous data of the target data in the paradigm Huffman tree and the node corresponding to the target data as the target coding of the target data. Since the node corresponding to the previous data has fewer layers than the root node, in most cases, the encoding of the path between the node corresponding to the previous data and the node corresponding to the target data is shorter than the encoding of the path between the root node and the node corresponding to the target data. By improving the Huffman coding, the compression efficiency of the Huffman coding is improved, and the amount of data after the data is compressed by the Huffman coding is reduced, so that more data can be temporarily stored in the buffer.
[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A multi-class data intelligent caching method, characterized in that: The method comprises the following steps: Convert each type of data to be cached in the buffer into a data sequence, obtain all types of data for each type of data, count the frequency of each type of data in the data sequence, sort all types of data in descending order of frequency as a data sequence sequence, and construct a paradigm Huffman tree based on the data sequence sequence and the binary tree; Encoding each data in the data sequence in sequence, including: taking the data to be encoded as target data, recording the node corresponding to the target data in the normalized Huffman tree as the target node, and recording the encoding of the shortest path from the root node to the target node as the first encoding of the target data; If the target data is the first data in the data sequence, the first code of the target data is used as the target code of the target data; otherwise, the previous data of the target data is recorded as the prefix data: In the data sequence, if the prefix data is before the target data, a second code of the target data is obtained according to a path of the prefix data and the target data in the normal form Huffman tree, and a target code of the target data is obtained according to the first code and the second code of the target data; if the prefix data is after the target data, a third code of the target data is obtained according to a path of the prefix data and the target data in the normal form Huffman tree, and a target code of the target data is obtained according to the first code and the third code of the target data; The target codes of all data in the data sequence are sequentially recorded as a coding sequence, and the coding sequence is the compression result of each type of data to be cached; The method of obtaining the target code of the target data according to the first code and the second code of the target data comprises the following specific steps: if the length of the first code of the target data is less than or equal to the length of the second code of the target data, using the first code as the target code of the target data; otherwise, using the second code as the target code of the target data; The method of obtaining the target code of the target data based on the first code and the third code of the target data includes the following specific steps: if the length of the first code is less than or equal to the length of the third code, the first code is used as the target code of the target data; otherwise, the third code is used as the target code of the target data.
2. A multi-class data intelligent caching method according to claim 1, characterized in that: The specific steps of constructing the paradigm Huffman tree are as follows: Construct an N+1-layer binary tree, requiring that the left nodes of all layers in the binary tree and the right nodes of the last layer are leaf nodes, the other nodes in the binary tree are parent nodes, and the node of the first layer in the binary tree is the root node, where N is the number of all kinds of data in the data sequence; the left nodes of all layers except the first two layers and the right nodes of the last layer are recorded as target nodes, and each data in the data sequence is assigned to each target node in turn in order from top to bottom, and the binary tree is assigned codes in the manner of left 1 and right 0, and the obtained binary tree is recorded as a normalized Huffman tree.
3. A multi-class data intelligent caching method according to claim 1, characterized in that: The specific steps of obtaining the second encoding of the target data include: The node corresponding to the prefix data in the normal form Huffman tree is recorded as the start node, the node corresponding to the target data in the normal form Huffman tree is recorded as the end node, and the code of the shortest path from the start node to the end node in the normal form Huffman tree is obtained and recorded as the second code.
4. A multi-class data intelligent caching method according to claim 1, characterized in that: The specific steps of obtaining the third encoding of the target data include: The node corresponding to the target data in the normal form Huffman tree is recorded as the start node, and the node corresponding to the prefix data in the normal form Huffman tree is recorded as the end node. The code of the shortest path from the start node to the end node is obtained in the normal form Huffman tree, and a 1 is added before the code as the third code.
5. A multi-class data intelligent caching system, characterized in that: It includes a cache partitioning unit, a data compression unit, and a data storage unit. The cache partitioning unit is used to query the type of data to be cached, set a corresponding identifier for each type of data, and divide the corresponding cache area for each type of data; the data compression unit implements the steps of a multi-type data intelligent caching method according to any one of claims 1 to 4, and the data storage unit is used to store the compression result of each type of data in the corresponding cache area.
Citation Information
Patent Citations
Methods and apparatus for rational compression and decompression of numbers
CN106301385A
Use of data prefixes to increase compression ratios
US20190273508A1