A data information processing method and system based on big data

Abstract

The present invention belongs to the field of data processing technology, and specifically relates to a data information processing method and system based on big data. The method includes: constructing a Huffman tree based on frequency statistics of all data in the data information; obtaining a special layer in which the increase in data volume is less than the decrease in data volume when the path code from the root node to the branch node is replaced with a fixed-length code of the sequence number of the branch node in the layer; and subsequently encoding all data through the Huffman tree and the special layer. For target data corresponding to the special layer, the branch node of the leaf node corresponding to the target data in the special layer is used as a guide node, and the fixed-length code of the guide node in the special layer is spliced with the path code from the guide node to the leaf node corresponding to the target data to obtain the encoding result of the target data. The present invention improves the compression rate of the data information compression result.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology. More specifically, the present invention relates to a data information processing method and system based on big data. Background Art

[0002] With the acceleration of digital transformation, enterprises and service providers are facing an unprecedented flood of data. From the Internet and the Internet of Things to internal enterprise systems, the speed and scale of data generation are growing exponentially. This rapidly growing amount of digital information not only requires more powerful data centers to process and analyze this data, but also requires enterprises and service providers to have greater storage capacity.

[0003] The "4V" model of big data - large data volume (Volume), diverse data types (Variety), fast processing speed (Velocity), and data authenticity (Veracity) - and data value (Value) together constitute the core characteristics of big data.

[0004] These characteristics give big data broad application prospects in various fields, but also bring challenges in data storage, processing and analysis.

[0005] Huffman Coding is widely used in data compression due to its simplicity and efficiency. It achieves lossless data compression by assigning shorter codes to data that occurs more frequently and longer codes to data that occurs less frequently.

[0006] By rationally utilizing the frequency information of the data, Huffman coding can significantly reduce the storage cost of data while ensuring data integrity. Therefore, the core of Huffman coding lies in the utilization of the frequency information of the data. At the same time, the efficiency of Huffman coding is also limited by the frequency information of the data, resulting in the inability to further improve the compression rate. Summary of the Invention

[0007] To solve the technical problem that the efficiency of Huffman coding is also limited by the frequency information of the data, resulting in an inability to further improve the compression rate, the present invention provides solutions in the following aspects.

[0008] In the first aspect, the present invention provides a data information processing method based on big data, including: constructing a Huffman tree according to the frequency statistics of all data in the data information; taking any layer in the Huffman tree that contains both leaf nodes and branch nodes as the target layer, and recording all data in the target layer where the layer of the leaf node corresponding to the previous data in the data information is located as identification data; taking all data in the row where the layer number of the corresponding leaf node in all identification data is greater than the layer number of the target layer as target data; determining the length of the fixed-length code of the branch node of the target layer according to the number of all branch nodes in the target layer; and determining the length of the fixed-length code of the branch node of the target layer according to the path from the root node to the branch node of the target layer. The reduction amount is determined by the length of the code and the length of the fixed-length code of the branch node of the target layer; in response to the number of identification data being less than the product of the number of target data and the reduction amount, the target layer is used as a special layer; all data are encoded through the Huffman tree and the special layer, wherein the encoding method for any target data corresponding to the special layer includes: using the branch node of the leaf node corresponding to the target data in the special layer as the guide node; determining the fixed-length code of the guide node according to the serial number of the guide node in the special layer; splicing the fixed-length code of the guide node with the path code from the guide node to the leaf node corresponding to the target data to obtain the encoding result of the target data.

[0009] The present invention obtains a special layer from the Huffman tree, in which the increase in data volume is less than the decrease in data volume when the path code from the root node to the branch node is replaced by a fixed-length code of the serial number of the branch node in the layer where it is located, thereby ensuring that when all data are subsequently encoded through the Huffman tree and the special layer, the compression rate of the compression result of the data information is improved; further, in the process of encoding all data through the Huffman tree, the present invention replaces the path code from the root node to the branch node of the target data in the special layer with a fixed-length code of the serial number of the branch node in the layer where it is located, thereby shortening the coding length of the target data and further improving the compression rate of the compression result of the data information.

[0010] Preferably, the determining of the length of the fixed-length code of the branch nodes of the target layer according to the number of all branch nodes in the target layer includes: recording the number of all branch nodes in the target layer as , then the length of the fixed-length code of the branch node of the target layer is equal to , Indicates rounding up. Represents the base-2 logarithm function.

[0011] Preferably, the reduction amount is determined based on the length of the path coding from the root node to the branch node of the target layer and the length of the fixed-length coding of the branch node of the target layer, including: the length of the path coding from the root node to the branch node of the target layer is equal to the layer number of the target layer minus 1; then the reduction amount is equal to the difference between the length of the path coding from the root node to the branch node of the target layer and the length of the fixed-length coding of the branch node of the target layer.

[0012] By calculating the reduction amount, the present invention can accurately determine whether the amount of data can be reduced when the target data corresponding to the target layer is subsequently encoded if the path encoding from the root node to the branch node is replaced by a fixed-length encoding of the sequence number of the branch node in the layer where it is located, and then determine a special layer that can improve the compression rate of the compression result of the data information.

[0013] Preferably, the encoding of all data through the Huffman tree and the special layer also includes: for any data in the data information, if the data is the identification data corresponding to the special layer but not the target data corresponding to the special layer, the path from the root node to the leaf node corresponding to the data is encoded as the encoding result of the data.

[0014] Preferably, the encoding of all data through the Huffman tree and the special layer also includes: for any data in the data information, if the data is not the identification data corresponding to the special layer, encoding the path from the root node to the leaf node corresponding to the data as the encoding result of the data.

[0015] Preferably, the method also includes: for any data in the data information: if the data is the target data corresponding to the special layer, add an identifier 1 in front of the encoding result of the data; if the data is the identification data corresponding to the special layer but not the target data corresponding to the special layer, add an identifier 0 in front of the encoding result of the data.

[0016] The present invention adds an identifier to distinguish whether to replace the path code from the root node to the branch node with the fixed-length code of the serial number of the branch node in the layer when encoding the subsequent data when the layer where the leaf node corresponding to the previous data is located is a special layer, thereby ensuring the accuracy of subsequent decoding.

[0017] Preferably, obtaining the frequency statistics of all the data includes: counting the number of times each type of data appears in the data information, wherein the same data belongs to the same type of data; the frequency of each type of data refers to the ratio of the number of times each type of data appears to the total number of all the data in the data information.

[0018] Preferably, the method further comprises: taking the frequency of all data as supplementary information and storing the information.

[0019] The present invention stores the frequencies of all data as supplementary information, thereby ensuring that the compression results of the data information can be correctly decoded later.

[0020] Preferably, the method further comprises: taking the layer numbers of all special layers as supplementary information and storing the information.

[0021] The present invention stores the layer numbers of all special layers as supplementary information, which can ensure that the compression results of the data information can be correctly decoded later.

[0022] In a second aspect, the present invention provides a data information processing system based on big data, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned data information processing method based on big data is implemented.

[0023] By adopting the above technical solution, the above-mentioned data information processing method based on big data is generated into a computer program and stored in the memory to be loaded and executed by the processor, so that a terminal device is made according to the memory and the processor for easy use.

[0024] The beneficial effects of the present invention are:

[0025] The present invention obtains a special layer from a Huffman tree, in which the increase in data volume is less than the decrease in data volume when the path code from the root node to the branch node is replaced by a fixed-length code of the sequence number of the branch node in the layer where it is located. In the subsequent process of encoding all data through the Huffman tree and the special layer, for any target data corresponding to the special layer, the path code from the root node to the branch node of the target data in the special layer is replaced by a fixed-length code of the sequence number of the branch node in the layer where it is located, thereby shortening the coding length of the target data and further improving the compression rate of the compression result of the data information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart schematically illustrating a data information processing method based on big data in the present invention;

[0027] Figure 2 is a diagram schematically illustrating a constructed Huffman tree. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0029] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0030] The embodiment of the present invention discloses a data information processing method based on big data, referring to Figure 1 , comprising steps S1-S3:

[0031] S1. Construct a Huffman tree based on the frequency statistics of all data in the data information.

[0032] Huffman coding is an algorithm widely used in data compression. It achieves lossless data compression by assigning shorter codes to data that appears more frequently and longer codes to data that appears less frequently.

[0033] The core idea of Huffman coding is to construct a Huffman tree based on the frequency of all data. The Huffman tree is a binary tree in which each leaf node represents a data and its frequency of occurrence. The Huffman tree construction process ensures that data with higher frequency of occurrence has shorter codes, while data with lower frequency of occurrence has longer codes. In this way, Huffman coding can effectively reduce the total length of the data and achieve data compression.

[0034] In order to construct a Huffman tree, it is necessary to first count the types of data in the data information and the frequency of each type of data; specifically, in the data information, count the number of times each type of data appears, where the same data belongs to the same type of data; accordingly, the frequency of each type of data refers to the ratio of the number of times each type of data appears to the total number of all data in the data information.

[0035] Furthermore, a Huffman tree is constructed based on the frequency of all data. The Huffman tree is a binary tree. Therefore, the node of the first layer of the Huffman tree is the root node, and the nodes of the remaining layers are divided into leaf nodes and branch nodes. Leaf nodes do not have subtrees, while branch nodes do have subtrees.

[0036] For example, for the data information of {5,6,1,3,7,4,5,8,6,6,6,5,8,6,6,8,1,6,6,5,8,1,5,6,6,8,6,5,1,3,7,2,6,1,3,6,5,8,1,8}, the total number of all data in the data information is 40, there are 8 kinds of data, namely: 1, 2, 3, 4, 5, 6, 7, 8, and the number of times each data appears is 6, 1, 3, 1, 7, 13, 2, 7, respectively. Then the frequency of each data is 0.15, 0.025, 0.075, 0.025, 0.175, 0.325, 0.05, and 0.175 respectively. According to the frequency of all data, the schematic diagram of the constructed Huffman tree is as follows Figure 2 shown.

[0037] In addition, in order to enable correct decoding, it is necessary to store the frequency information of each data. Therefore, the frequency of all data is stored as supplementary information.

[0038] S2. Get all special layers in the Huffman tree.

[0039] In the process of encoding all data through the Huffman tree, if the layer where the leaf node corresponding to the previous data is located (denoted as Layer) in the layer where the leaf node corresponding to the next data is located (recorded as layer), i.e. , then the subtree where the next data is located is at the layer where the leaf node corresponding to the previous data is located, that is, There is a corresponding branch node in each layer; for example, if the previous data is 6, the layer where data 6 is located is the 3rd layer, and the next data is 1, the layer where data 1 is located is the 4th layer, the subtree where data 1 is located has a corresponding branch node in the layer where data 6 is located, that is, the 3rd layer, that is, the branch node with a frequency of 0.325 in the 3rd layer.

[0040] Conventional Huffman coding encodes the path from the root node to the next data as the encoding result of the next data. During subsequent decoding, the path on the Huffman tree is determined based on the encoding result, and then the next data is determined based on the path. For example, the encoding result of data 1 is 100.

[0041] Since the layer where the leaf node corresponding to the previous data is located is above the layer where the leaf node corresponding to the next data is located, the subtree where the next data is located has a corresponding branch node in the layer where the leaf node corresponding to the previous data is located. In other words, as long as the previous data is known, the layer where a branch node of the subtree where the next data is located can be known based on the layer where the leaf node corresponding to the previous data is located. On this basis, it is only necessary to record the serial number of the branch node in the layer where the leaf node corresponding to the previous data is located to determine the position of the branch node. Then, based on the position of the branch node and the path encoding from the branch node to the next data, the path on the Huffman tree can be determined, and then the next data can be determined based on the path.

[0042] Therefore, the path encoding from the root node to the next data is split into the path encoding from the root node to the branch node (the branch node corresponding to the layer where the next data is located at the leaf node corresponding to the previous data) and the path encoding from the branch node to the next data, where the length of the path encoding from the root node to the branch node is equal to , Indicates the number of layers where the branch node is located, that is, the number of layers where the leaf node corresponding to the previous data is located; at this time, through the branch node in the The fixed-length code of the layer number is used to replace the path code from the root node to the branch node, where the branch node is in the first The fixed-length encoding length of the layer number is equal to , Indicates the The number of all branch nodes in the layer, that is, as long as the The number of all branch nodes in the layer satisfy , the encoding length of the latter data can be shortened by replacing the data.

[0043] For example, when the previous data is 6 and the next data is 1, the layer where the leaf node corresponding to the previous data is located (layer 3) is above the layer where the leaf node corresponding to the next data is located (layer 4). At this time, the subtree where the next data 1 is located has a corresponding branch node in the layer where the previous data 6 is located, that is, the layer 3, that is, the branch node with a frequency of 0.325 in the layer 3; since there is only one branch node in the layer 3, there is no need to encode the sequence number of the branch node, that is, the length of the fixed-length code of the sequence number of the branch node in the layer 3 is equal to , represents the number of all branch nodes in layer 3, and ; At this time, the number of all branch nodes in the third layer satisfy Therefore, through the replacement operation, the encoding length of the latter data can be shortened.

[0044] In summary, in the subsequent process of encoding data through the Huffman tree, if you want to shorten the encoding length of the data by replacing the path encoding from the root node to the branch node with a fixed-length encoding of the branch node's sequence number at the layer where it is located, the data must meet two conditions:

[0045] Condition 1: The layer where the leaf node corresponding to the previous data of the data is located (the The number of all branch nodes in the layer satisfy Among them, due to the The layer is the layer where the leaf node corresponding to the previous data is located. In other words, condition 1 also contains a hidden condition, which requires the first There must be leaf nodes in the layer.

[0046] Condition 2: The layer where the leaf node corresponding to the previous data of the data is located (the Layer) The leaf node corresponding to the data is located in the layer ( layer), i.e. ; Among them, since the leaf node corresponding to the previous data is located in the layer ( Layer) The leaf node corresponding to the data is located in the layer ( That is, condition 2 also contains a hidden condition, which requires the first There must be branch nodes in the layer.

[0047] For the hidden conditions in conditions 1 and 2, only the sequence number of the layer and the number of leaf nodes and branch nodes in the layer are involved. Therefore, the candidate layer that meets the conditions can be obtained directly by judging each layer of the Huffman tree: it is required that there are branch nodes and leaf nodes in the candidate layer, and the number of all branch nodes in the candidate layer is satisfy .

[0048] Specifically, for the constructed Huffman tree, for the first layer, obtain the The number of all leaf nodes and the number of all branch nodes in the layer are used to determine whether to The layer is used as an alternative layer. The specific operation process is as follows:

[0049] 1. If There are both leaf nodes and branch nodes in the layer, and the The number of all branch nodes in the layer satisfy , will Layer as an alternative layer, Indicates rounding up.

[0050] 2. If There are no leaf nodes in the layer, or There is no branch node in the layer, or The number of all branch nodes in the layer satisfy , then you cannot layer as an alternative layer.

[0051] For example, for Figure 2 The Huffman tree shown in the figure, the process of obtaining all candidate layers in the Huffman tree is:

[0052] (1) Since there are no leaf nodes in the first and second layers, they cannot be used as alternative layers.

[0053] (2) Since there are both leaf nodes and branch nodes in the third layer, and the number of all branch nodes in the fourth layer is =1, then , therefore, the number of all branch nodes in layer 3 satisfy , the third layer is used as an alternative layer.

[0054] (3) Similarly, since there are leaf nodes and branch nodes in the 4th, 5th and 6th layers, and the number of all branch nodes in the 4th layer is =1, the number of all branch nodes in the 5th layer =1, the number of all branch nodes in the 6th layer =1, , , , therefore, the number of all branch nodes in the 4th, 5th and 6th layers is 、 and All satisfied , then the 4th, 5th and 6th layers are taken as candidate layers.

[0055] (4) Since there is no branch node in the 7th layer, the 7th layer cannot be used as an alternative layer.

[0056] It should be noted that, by performing a first screening of all layers based on the number of branch nodes, thereby screening out candidate layers, the number of layers that need to be judged and compared when subsequently obtaining special layers can be reduced, thereby improving data compression efficiency.

[0057] In addition, when the layer where the leaf node corresponding to the previous data of the data is located is an alternative layer, it is necessary to use an identifier to identify whether the layer where the leaf node corresponding to the previous data of the data is located is above the layer where the leaf node corresponding to the data is located, so as to further determine whether to shorten the encoding length of the data by replacing the path encoding from the root node to the branch node with a fixed-length encoding of the serial number of the branch node in the layer; since there are only two cases: the layer where the leaf node corresponding to the previous data of the data is located is above the layer where the leaf node corresponding to the data is located or the layer where the leaf node corresponding to the previous data of the data is not above the layer where the leaf node corresponding to the data is located, therefore, it is only necessary to introduce an identifier with a length of 1: when the layer where the leaf node corresponding to the previous data of the data is located is above the layer where the leaf node corresponding to the data is located, that is, When the identifier is 1, when the layer where the leaf node corresponding to the previous data of the data is located is not above the layer where the leaf node corresponding to the data is located, that is, , the identifier is 0.

[0058] Therefore, when encoding the data information later, the layer where the leaf node corresponding to the previous data is located (the Layer) is the data of the alternative layer. If the path code from the root node to the branch node is replaced by the fixed-length code of the branch node's sequence number in the layer, an identifier with a length of 1 needs to be added. This will increase the amount of data in the final data information encoding compression result, and the amount of data increase is equal to the number of all data in the layer where the leaf node corresponding to the previous data is the alternative layer multiplied by the length of the identifier, that is, the amount of data increase is equal to , The layer where the leaf node corresponding to the previous data in the data information is located is the candidate layer (the The number of all data layers.

[0059] In addition, for all data in the candidate layer where the leaf node corresponding to the previous data is located, only the layer where the leaf node corresponding to the previous data is located (the Layer) The leaf node corresponding to the data is located in the layer ( layer), i.e. When the root node is connected to the data The path code of the branch node in the first layer is replaced by the fixed-length code of the branch node’s serial number in the layer to shorten the encoding length of the data. The length of the path encoding of the branch node of the layer is equal to , the data is in The length of the fixed-length code of the sequence number of the branch node of the layer is equal to , For the The number of all branch nodes in the layer, so the reduction in the encoding length of the data is equal to ; In summary, the alternative layer ( layer) as a special layer, the amount of data reduction is equal to the leaf node corresponding to the previous data. The layer where the leaf node is located is the alternative layer (the All data in the layer) that meet The number of all data Multiply by the reduction in encoding length of each data , that is, the reduction in data volume is equal to .

[0060] In summary, if the amount of data increases Greater than or equal to the reduction By replacing the path code from the root node to the branch node with the fixed-length code of the branch node's sequence number at the layer where it is located, the length of the encoded result of the data information cannot be shortened; only the amount of data increases. Less than the reduction When the path code from the root node to the branch node is replaced by the fixed-length code of the branch node's sequence number at the layer where it is located, the length of the encoded result of the data information can be shortened; therefore, the increase in the amount of data will be met. Less than the reduction The alternative layer is used as a special layer. For the data where the leaf node corresponding to the previous data is in a special layer, it will be considered to replace the path code from the root node to the branch node with the fixed-length code of the sequence number of the branch node in the layer.

[0061] Specifically, for layer number The specific operation process of determining whether to use the candidate layer as a special layer is as follows:

[0062] 1. The leaf node corresponding to the previous data in the obtained data information is located at the first layer. All data of the layer are recorded as identification data; the number of all identification data is recorded as .

[0063] 2. Set the layer number of the row to be greater than All identification data are recorded as target data, and the number of all target data is recorded as .

[0064] 3. If Less than , indicating that When the layer is used as a special layer, the increase in data volume is less than the decrease in data volume. Therefore, Layer as a special layer, where For the The number of all branch nodes in the layer.

[0065] 4. If Greater than or equal to , indicating that When the layer is used as a special layer, the increase in data volume is greater than or equal to the decrease in data volume. Therefore, the first layer cannot be used as a special layer. layer as a special layer.

[0066] For example, when the data information is {5,6,1,3,7,4,5,8,6,6,6,5,8,6,6,8,1,6,6,5,8,1,5,6,6,8,6,5,1,3,7,2,6,1,3,6,5,8,1,8}, for Figure 2 In the Huffman tree shown in FIG, all candidate layers in the obtained Huffman tree include the 3rd layer, the 4th layer, the 5th layer, and the 6th layer. The process of determining whether to use the candidate layer as a special layer is as follows:

[0067] (1) For the third layer, there are 3 leaf nodes and 1 branch node. The data corresponding to the 3 leaf nodes are data 5, data 6, and data 8 respectively. The layer where the leaf node corresponding to the previous data in the obtained data information is located is all the data in the third layer, that is, the previous data is all the data of data 5 / data 6 / data 8, which is 26 in total and recorded as identification data. Then the number of all identification data is =26; Among these 26 identification data, there are 6 target data that meet the layer number greater than 3 in the row, then =6; the third layer includes 1 branch node, then =1, accordingly, =12; due to =26, so Greater than , which means that when Layer 3 is used as a special layer, the increase in data volume is greater than the decrease in data volume. Therefore, Layer 3 cannot be used as a special layer.

[0068] (2) For the 4th layer, there are 1 leaf node and 1 branch node. The data corresponding to the leaf node is data 1. The layer where the leaf node corresponding to the previous data in the data information is located is all the data in the 4th layer, that is, all the data whose previous data is data 1. There are 6 of them in total and they are recorded as identification data. Then the number of all identification data is =6; Among these 6 identification data, there are 3 target data that satisfy the layer number of the row greater than 4, then =3; the fourth layer includes 1 branch node, then =1, accordingly, =9; due to =6, so Less than , which means that when the 4th layer is used as a special layer, the increase in data volume is less than the decrease in data volume. Therefore, the 4th layer is used as a special layer.

[0069] (3) Similarly, for the fifth layer, , , ,therefore, =8, Less than , indicating that the increase in data volume when the 5th layer is used as a special layer is less than the decrease in data volume. Therefore, the 5th layer is used as a special layer. For the 6th layer, , , ,therefore, =10, Less than , which means that when the 6th layer is used as a special layer, the increase in data volume is less than the decrease in data volume. Therefore, the 6th layer is used as a special layer.

[0070] In addition, in order to enable correct decoding, the layer number of each special layer needs to be stored. Therefore, the layer numbers of all special layers are stored as supplementary information.

[0071] S3. Encode all data according to the Huffman tree and the special layer in the Huffman tree to obtain the compressed result of the data information and store it.

[0072] Huffman coding is a path coding from the root node of the Huffman tree to the leaf node corresponding to each data, and the data is encoded and compressed. Therefore, it is necessary to assign codes of 0 and 1 to the paths in the Huffman tree. Therefore, in one embodiment, starting from the root node of the Huffman tree, the code "0" is assigned to the left subtree and the code "1" is assigned to the right subtree until the leaf node is reached. In another embodiment, starting from the root node of the Huffman tree, the code "1" is assigned to the left subtree and the code "0" is assigned to the right subtree until the leaf node is reached.

[0073] Furthermore, all data are encoded through the Huffman tree. During the encoding process, for any data:

[0074] (1) If the data is the target data corresponding to the special layer, the branch node of the leaf node corresponding to the data in the special layer is used as the guide node; the fixed-length code of the guide node is determined according to the sequence number of the guide node in the special layer; the fixed-length code of the guide node is concatenated with the path code from the guide node to the leaf node corresponding to the target data to obtain the encoding result of the target data; at the same time, an identifier 1 is added in front of the encoding result of the data.

[0075] For example, when the previous data is 1 and the next data is 3, the leaf node corresponding to the previous data is The layer is the 4th layer, and the leaf node corresponding to the next data The layer is the 5th layer; because the leaf node corresponding to the previous data The layer where it is located, that is, the 4th layer, is a special layer. Therefore, determine the leaf node corresponding to the previous data. The leaf node corresponding to the layer and the next data The size relationship of the layer number of the layer, because the leaf node corresponding to the previous data The layer number is 4, and the leaf node corresponding to the next data The layer number is 5, 5>4, so the latter data is the target data corresponding to the special layer, and the leaf node is obtained. The branch node corresponding to the subtree in the 4th layer, that is, the branch node with a frequency of 0.175 in the 4th layer, is recorded as the guide node ; Since there is only one branch node in the fourth layer, there is no need to Encode the serial number; get the boot node To the leaf node Path encoding If it is 0, the encoding result of the next data is 0, and an identifier 1 is added in front of the encoding result of the next data. Therefore, the encoding result of data 6 is 01.

[0076] (2) If the data is the identification data corresponding to the special layer but not the target data corresponding to the special layer, encode the path from the root node to the leaf node corresponding to the data as the encoding result of the data; at the same time, add the identifier 0 in front of the encoding result of the data.

[0077] For example, when the previous data is 1 and the next data is 5, the leaf node corresponding to the previous data is The layer is the 4th layer, and the leaf node corresponding to the next data The layer is the 3rd layer; because the leaf node corresponding to the previous data The layer where it is located, that is, the 4th layer, is a special layer. Therefore, determine the leaf node corresponding to the previous data. The leaf node corresponding to the layer and the next data The size relationship of the layer number of the layer, because the leaf node corresponding to the previous data The layer number is 4, and the leaf node corresponding to the next data The layer number is 3, 3<4, so the latter data is the identification data corresponding to the special layer but not the target data corresponding to the special layer. The path encoding is used as the encoding result of the next data, then the encoding result of data 6 is 11, and the identifier 0 is added in front of the encoding result of the next data, so the encoding result of data 6 is 011.

[0078] (3) If the data is not the identification data corresponding to the special layer, encode the path from the root node to the leaf node corresponding to the data as the encoding result of the data.

[0079] For example, when the previous data is 5 and the next data is 6, the leaf node corresponding to the previous data is The layer is the third layer. Since the third layer is not a special layer, directly connect the root node to the leaf node. The path encoding of is used as the encoding result of the next data, and the encoding result of data 6 is 11.

[0080] Exemplarily, when the data information is {5, 6, 1, 3, 7, 4, 5, 8, 6, 6, 5, 8, 6, 6, 8, 1, 6, 6, 5, 8, 1, 5, 6, 6, 8, 6, 5, 1, 3, 7, 2, 6, 1, 3, 6, 5, 8, 1, 8}, the data information is encoded by conventional Huffman coding and the method of this embodiment, and the results are:

[0081] (1) The data information is encoded by conventional Huffman coding, and the encoding results of each data are 00, 11, 100, 1010, 10111, 101101, 00, 01, 11, 11, 11, 00, 01, 11, 11, 01, 100, 11, 11, 00, 01, 100, 00, 11, 11, 01, 11, 00, 100, 1010, 10111, 101100, 11, 10 0, 1010, 11, 00, 01, 100, 01. The encoding result of the data information is 0011100101010111101101000111111100011111011001111000110000111101110010010101011110110011100101011000110001. The length of the encoding result of the data information is 106.

[0082] (2) The data information is encoded by the method of this embodiment, and the encoding results of each data are 00, 11, 100, 10, 11, 11, 00, 01, 11, 11, 00, 01, 11, 11, 01, 100, 011, 11, 00, 01, 100, 000, 11, 11, 01, 11, 00, 100, 10, 11, 10, 11, 10 0, 10, 011, 00, 01, 100, 001. The encoding result of the data information is 001110010111100011111110001111101100011110001100000111101110010010111011100100110001100001. The length of the encoding result of the data information is 90.

[0083] (3) In summary, when the data information is encoded using conventional Huffman coding, the length of the encoded result of the data information is equal to 106; when the data information is encoded using the method of this embodiment, the length of the encoded result of the data information is equal to 90; therefore, the method of this embodiment can improve the compression rate of the compression result of the data information.

[0084] It should be noted that in the process of compressing and encoding data information through Huffman coding, a special layer is obtained from the Huffman tree by replacing the path code from the root node to the branch node with a fixed-length code of the serial number of the branch node in the layer where the increase in data volume is less than the decrease in data volume, thereby ensuring that when all data are subsequently encoded through the Huffman tree and the special layer, for the next data that satisfies the requirement that the layer where the leaf node corresponding to the previous data is located is a special layer and the layer number of the layer where the corresponding leaf node is located is greater than the layer number of the special layer, when encoding, the path code from the root node to the branch node of the next data in the special layer is replaced with a fixed-length code of the serial number of the branch node in the layer where the branch node is located, thereby shortening the coding length of the next data and thereby improving the compression rate of the compression result of the data information.

[0085] Furthermore, when the data information needs to be viewed or used, the compressed result of the data information is decoded. The specific process is as follows:

[0086] 1. Construct a Huffman tree based on the frequency of all data stored in the supplementary information.

[0087] 2. According to the Huffman tree and the special layer in the Huffman tree, the compressed result of the data information is decoded. During the decoding process, the data that has just been decoded is recorded as the previous data, and the next data to be decoded is recorded as the next data; the leaf node corresponding to the previous data is obtained. The layer where it is located is recorded as Layer, the specific decoding process is:

[0088] 2.1、Judge Is the layer a special layer? The layer is not a special layer. The path code is obtained directly based on the remaining compression results and the Huffman tree, and the leaf node is obtained based on the path code. , the leaf nodes The corresponding data is used as the latter data.

[0089] 2.2, if The layer is a special layer. The first binary data in the remaining compression result is judged to be 0 or 1 to determine whether to replace the path code from the root node to the branch node with the fixed-length code of the branch node's sequence number in the layer during encoding:

[0090] (1) If it is 0, it means that during encoding, the path code from the root node to the branch node is not replaced by the fixed-length code of the branch node's serial number in the layer where it is located. The path code is directly obtained based on the remaining compression result and the Huffman tree, and the leaf node is obtained based on the path code. , the leaf nodes The corresponding data is used as the latter data.

[0091] (2) If it is 0, it means that when encoding, the path code from the root node to the branch node is replaced by the fixed-length code of the branch node's serial number in the layer where it is located; first obtain the first The number of all branch nodes in the layer , and then obtain a length of The binary number, the decimal number corresponding to the binary number is used as the serial number to obtain the branch node; according to the branch node and the remaining compression result, the path code is obtained from the Huffman tree, and the leaf node is obtained according to the path code , the leaf nodes The corresponding data is used as the latter data.

[0092] An embodiment of the present invention further discloses a data information processing system based on big data, including a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, a data information processing method based on big data according to the present invention is implemented.

[0093] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

Claims

1. A data information processing method based on big data, characterized in that: include: Construct a Huffman tree based on the frequency statistics of all data in the data information; Any layer in the Huffman tree that contains both leaf nodes and branch nodes is taken as the target layer, and all data in the data information whose leaf node corresponding to the previous data is in the target layer are recorded as identification data; all data in the row where the layer number of the corresponding leaf node in all identification data is greater than the layer number of the target layer are taken as target data; the length of the fixed-length code of the branch node of the target layer is determined according to the number of all branch nodes in the target layer; determining a reduction amount based on a length of a path code from a root node to a branch node of a target layer and a length of a fixed-length code of the branch node of the target layer; and in response to the number of identification data being less than a product of the number of target data and the reduction amount, treating the target layer as a special layer; All data are encoded through the Huffman tree and the special layer, wherein the encoding method for any target data corresponding to the special layer includes: using the branch node of the leaf node corresponding to the target data in the special layer as the guide node; determining the fixed-length code of the guide node according to the sequence number of the guide node in the special layer; splicing the fixed-length code of the guide node with the path code from the guide node to the leaf node corresponding to the target data to obtain the encoding result of the target data.

2. The data information processing method based on big data according to claim 1, characterized in that: The determining, according to the number of all branch nodes in the target layer, the length of the fixed-length code of the branch nodes of the target layer includes: The number of all branch nodes in the target layer is recorded as , then the length of the fixed-length code of the branch node of the target layer is equal to , Indicates rounding up. Represents the base-2 logarithm function.

3. The data information processing method based on big data according to claim 1, characterized in that: The determining the reduction amount according to the length of the path code from the root node to the branch node of the target layer and the length of the fixed-length code of the branch node of the target layer includes: The length of the path code from the root node to the branch node of the target layer is equal to the layer number of the target layer minus 1; The reduction amount is equal to the difference between the length of the path code from the root node to the branch node of the target layer and the length of the fixed-length code of the branch node of the target layer.

4. The data information processing method based on big data according to claim 1, characterized in that: The encoding of all data by the Huffman tree and the special layer also includes: For any data in the data information, if the data is identification data corresponding to the special layer but not target data corresponding to the special layer, the path from the root node to the leaf node corresponding to the data is encoded as the encoding result of the data.

5. The data information processing method based on big data according to claim 1, characterized in that: The encoding of all data by the Huffman tree and the special layer also includes: For any data in the data information, if the data is not the identification data corresponding to the special layer, the path from the root node to the leaf node corresponding to the data is encoded as the encoding result of the data.

6. The data information processing method based on big data according to claim 1, characterized in that: The method further comprises: For any data in the data information: if the data is the target data corresponding to the special layer, add an identifier 1 in front of the encoding result of the data; if the data is the identification data corresponding to the special layer but not the target data corresponding to the special layer, add an identifier 0 in front of the encoding result of the data.

7. The data information processing method based on big data according to claim 1, characterized in that: Obtain frequency statistics of all the data, including: In the data information, count the number of times each type of data appears, where the same data belongs to the same type of data; The frequency of each data type refers to the ratio of the number of times each data type appears to the total number of all data types in the data information.

8. The data information processing method based on big data according to claim 1, characterized in that: The method further includes: taking the frequencies of all data as supplementary information and storing the information.

9. The data information processing method based on big data according to claim 1, characterized in that: The method further includes: taking the layer numbers of all special layers as supplementary information and storing the information.

10. A data information processing system based on big data, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a data information processing method based on big data according to any one of claims 1 to 9 is implemented.

Images