Big data-based judicial data processing method and system

Abstract

The invention relates to the technical field of data coding, in particular to a judicial data processing method and system based on big data, and the method comprises the steps: collecting judicial data, converting the judicial data into a to-be-compressed sequence, obtaining a plurality of associated characters of each character in the to-be-compressed sequence, setting m virtual nodes and a plurality of real nodes, under different values of m, obtaining a compression advantage according to the predicted frequency of each virtual node and the predicted frequency of each real node, determining the value of m when the compression advantage is maximum as a target value, and under the target value, constructing a Huffman tree according to the predicted frequency of each virtual node and the predicted frequency of each real node, and compressing the to-be-compressed sequence according to the Huffman tree to obtain a compression result, and transmitting the compression result. The method is high in judicial data compression efficiency, so that the judicial data transmission efficiency is improved, and the storage cost is reduced.

Description

Technical Field

[0001] The present invention generally relates to the field of data coding technology. More specifically, the present invention relates to a judicial data processing method and system based on big data. Background Art

[0002] In the judicial system, judicial data needs to be transmitted to a central database or cloud server for analysis and management. Judicial data usually contains a large amount of text information, test results, equipment operation records, etc., which need to be compressed and transmitted.

[0003] Traditional Huffman coding constructs a Huffman tree for compression based on the frequency of each character. If Huffman coding is used to compress judicial data, the compression efficiency is limited by the frequency of each character in the judicial data and is difficult to further improve, resulting in low judicial data transmission efficiency, large storage space occupation, and is not conducive to the management of judicial data. Summary of the invention

[0004] In order to solve one or more of the above technical problems, the present invention proposes a judicial data processing method and system based on big data, and the method includes the following steps: Collect judicial data and convert the judicial data into sequences to be compressed; Obtain multiple associated characters for each character in the sequence to be compressed; Take each character as a real node; set m virtual nodes, where m corresponds to multiple different values; Under each value of m, the predicted frequency of each virtual node and the predicted frequency of each real node are determined according to the number of associated characters of each character and the number of associated characters of each character; the compression advantage is obtained according to the predicted frequency of each virtual node and the predicted frequency of each real node; The value of m when the compression advantage is the largest is taken as the target value of m; Under the target value, a Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node; the sequence to be compressed is compressed according to the code words corresponding to the virtual nodes and the real nodes in the Huffman tree to obtain a compression result; The compressed result is transmitted.

[0005] Preferably, the step of obtaining a plurality of associated characters for each character in the sequence to be compressed includes: Any character in the sequence to be compressed is used as the target character. When a character in the sequence to be compressed appears at the next position of the target character, this character is said to be associated with the target character. The number of each character associated with the target character when it is associated with the target character is counted. Each character associated with the target character is sorted in descending order of the number of each character associated with the target character. Each sorted character is sequentially used as the first associated character, the second associated character, the third associated character, ..., the eth associated character of the target character, where e represents the number of types of characters associated with the target character.

[0006] Preferably, m corresponds to multiple different values, including: m traverses and takes [1, ], where S represents the number of character types in the sequence to be compressed.

[0007] Preferably, the predicted frequency of the virtual node satisfies the expression: ; in, represents the predicted frequency of the i-th virtual node, i represents the sequence number of the virtual node, i takes every integer in [1,m], and m represents the number of virtual nodes; Indicates the length of the sequence to be compressed; Indicates the number of occurrences of the i-th associated character of the k-th character in the sequence to be compressed, where k represents the sequence number of the character. When the i-th associated character of the k-th character does not exist, ; S represents the number of character types in the sequence to be compressed.

[0008] Preferably, the predicted frequency of the real node satisfies the expression: ; in, represents the predicted frequency of the jth real node, j represents the sequence number of the real node, j takes every integer in [1, S], and S represents the number of character types in the sequence to be compressed; Indicates the frequency of the character corresponding to the jth real node in the sequence to be compressed; Indicates the length of the sequence to be compressed; represents a function of the i-th associated character of the k-th character and the character corresponding to the j-th real node. When the character corresponding to the j-th real node is different from the i-th associated character of the k-th character, is 0, when the character corresponding to the jth real node is the same as the i-th associated character of the kth character, is the number of occurrences of the i-th associated character of the k-th character associated with the k-th character. When the i-th associated character of the k-th character does not exist, is 0.

[0009] Preferably, the compression advantage satisfies the expression: ; in, Indicates compression advantage; represents the predicted frequency of the i-th virtual node, i represents the sequence number of the virtual node; m represents the number of virtual nodes; represents the predicted frequency of the jth real node, and j represents the sequence number of the real node; Represents the number of character types in the sequence to be compressed; exp() represents an exponential function with the natural constant e as the base.

[0010] Preferably, constructing a Huffman tree according to the predicted frequency of each virtual node and the predicted frequency of each real node includes: A Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node, and each virtual node and each real node respectively corresponds to a node in the Huffman tree.

[0011] Preferably, compressing the sequence to be compressed according to the code words corresponding to the virtual nodes and the real nodes in the Huffman tree to obtain the compression result includes: Each character in the sequence to be compressed is treated as a character to be encoded, and the characters to be encoded are compressed in sequence, specifically: When the character to be encoded is the first element in the sequence to be compressed, the codeword corresponding to the real node corresponding to the character to be encoded in the Huffman tree is used as the encoding result of the character to be encoded; When the character to be encoded is not the first element in the sequence to be compressed, determine whether the character to be encoded is the i-th associated character of the previous character, where i takes every integer in [1,M] and M represents the target value of m; when the character to be encoded is not the i-th associated character of the previous character, the code word corresponding to the real node corresponding to the character to be encoded in the Huffman tree is used as the encoding result of the character to be encoded; when the character to be encoded is the i-th associated character of the previous character, the code word corresponding to the i-th virtual node in the Huffman tree is used as the encoding result of the character to be encoded; The encoding results of all the characters to be encoded are concatenated together in the order of the characters to be encoded to obtain a compression result.

[0012] Preferably, the converting of judicial data into a sequence to be compressed includes: The judicial data is encoded into a binary sequence, and the binary sequence is encoded using Base64 to obtain a sequence to be compressed.

[0013] A judicial data processing system based on big data includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned judicial data processing method based on big data is implemented.

[0014] The present invention has the following beneficial effects: the present invention sets m virtual nodes, and determines the predicted frequency of each virtual node according to the number of associated characters of each character and the associated appearance of each character. Since the virtual node corresponds to the associated character of each character, the predicted frequency of the virtual node is relatively large, so that in the Huffman tree, the codeword corresponding to the virtual node is relatively short. In the process of compressing the sequence to be compressed, a large number of characters are encoded as codewords corresponding to the virtual nodes, so that the data volume of the final compression result is small, the compression efficiency of the judicial data is improved, and then the transmission efficiency of the judicial data is improved, and the storage cost of the judicial data is reduced.

[0015] Furthermore, the present invention sets multiple different values ​​for m. Under each value of m, the compression advantage is obtained according to the predicted frequency of each virtual node and the predicted frequency of each real node. The value of m when the compression advantage is the largest is used as the target value of m. A Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node under the target value, and the sequence to be compressed is compressed, thereby further improving the compression efficiency of judicial data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 It is a flowchart of the steps of the judicial data processing method based on big data in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] See also Figure 1, which shows a flowchart of a method for processing judicial data based on big data provided by an embodiment of the present invention, the method comprising the following steps: S001. Collect judicial data and convert the judicial data into sequences to be compressed.

[0020] For each judicial case, data such as the name, number, name of the person, and judicial result of the judicial items are recorded, collectively referred to as judicial data.

[0021] It should be noted that judicial data includes multiple data types, such as text and digital forms. Judicial data including multiple data types are difficult to be compressed uniformly. In order to facilitate subsequent compression, the present invention converts the judicial data into the same data type for compression.

[0022] In one embodiment, the judicial data is encoded into binary form using a first encoding algorithm to obtain a binary sequence, and the binary sequence is encoded using Base64 to obtain a sequence to be compressed. It should be noted that the present invention does not limit the first encoding algorithm, and the implementer can set the first encoding algorithm according to the actual implementation situation, such as GB2312 encoding and UTF-8 encoding. Base64 can encode binary into characters, so the obtained sequence to be compressed consists of characters.

[0023] S002. Obtain multiple associated characters of each character in the sequence to be compressed, set m virtual nodes and multiple real nodes, obtain compression advantage according to the predicted frequency of each virtual node and the predicted frequency of each real node under different values ​​of m, and determine the value of m when the compression advantage is the largest as the target value.

[0024] In one embodiment, any character in the sequence to be compressed is used as the target character. When a character in the sequence to be compressed appears at the next position of the target character, the character is said to be associated with the target character. The number of each character associated with the target character when it appears in association with the target character is counted. Each character associated with the target character is sorted in descending order of the number of each character associated with the target character. When the number of two characters associated with the target character is the same, they are sorted in the order of the two characters associated with the target character. Each sorted character is sequentially used as the first associated character, the second associated character, the third associated character, ..., the eth associated character of the target character, where e represents the number of types of characters associated with the target character.

[0025] For example, when the sequence to be compressed is {a,b,a,b,a,a,b,a,c,b,a,c,d}, the characters associated with character a are b,a,c. The number of times character b is associated with character a is 3, the number of times character a is associated with character a is 1, and the number of times character c is associated with character a is 2. Sort b,a,c by the number of times they are associated with character a, and the result is b,c,a. Then the first associated character of character a is b, the second associated character is c, and the third associated character is a.

[0026] Similarly, get all the associated characters for each character.

[0027] Set m virtual nodes, where m represents the number of virtual nodes. Treat each character as a real node.

[0028] Get the predicted frequency for each virtual node: ; in, represents the predicted frequency of the i-th virtual node, i represents the sequence number of the virtual node, i takes every integer in [1,m], and m represents the number of virtual nodes; Indicates the length of the sequence to be compressed; Indicates the number of occurrences of the i-th associated character of the k-th character in the sequence to be compressed, where k represents the sequence number of the character. When the i-th associated character of the k-th character does not exist, ; S represents the number of character types in the sequence to be compressed; Represents the total number of occurrences of the i-th associated characters of all different characters associated with the corresponding characters; the i-th associated characters of all different characters together constitute the i-th virtual node, so the ratio of the total number of occurrences of the i-th associated characters of all different characters associated with the corresponding characters to all characters in the sequence to be compressed is taken as the predicted frequency of the i-th virtual node.

[0029] Get the predicted frequency for each real node: ; in, represents the predicted frequency of the jth real node, j represents the sequence number of the real node, j takes every integer in [1, S], and S represents the number of character types in the sequence to be compressed; Indicates the frequency of the character corresponding to the jth real node in the sequence to be compressed; Indicates the length of the sequence to be compressed; represents a function of the i-th associated character of the k-th character and the character corresponding to the j-th real node. When the character corresponding to the j-th real node is different from the i-th associated character of the k-th character, is 0, when the character corresponding to the jth real node is the same as the i-th associated character of the kth character, is the number of occurrences of the i-th associated character of the k-th character associated with the k-th character; when the i-th associated character of the k-th character does not exist, is 0; Indicates the frequency of the character corresponding to the jth real node appearing as the first to the mth associated characters of the character in the sequence to be compressed. When the character corresponding to the jth real node appears as the first to the mth associated characters of the character in the sequence to be compressed, it has participated in the statistics of the predicted frequency of the virtual node. Therefore, it is necessary to count the frequency of the character corresponding to the jth real node appearing as the first to the mth associated characters of the character in the sequence to be compressed. Eliminate it to obtain the predicted frequency of the jth real node.

[0030] Get the compression benefit based on the predicted frequency of each virtual node and the predicted frequency of each real node: ; in, Indicates compression advantage; represents the predicted frequency of the i-th virtual node, i represents the sequence number of the virtual node; m represents the number of virtual nodes; represents the predicted frequency of the jth real node, and j represents the sequence number of the real node; Indicates the number of character types in the sequence to be compressed; exp() represents an exponential function with the natural constant e as the base; The information entropy of the set consisting of the predicted frequencies of all virtual nodes and all real nodes. If all virtual nodes and real nodes are regarded as a node in the Huffman tree, the average coding length corresponding to each node is approximately equal to the information entropy. When the information entropy is larger, the average coding length is longer, the corresponding compression efficiency is lower, and the compression advantage is smaller; conversely, when the information entropy is smaller, the average coding length is shorter, the corresponding compression efficiency is higher, and the compression advantage is greater.

[0031] Set the value range of m to [1, ], where S represents the number of characters in the sequence to be compressed. m traverses [1, ], obtain the compression advantage corresponding to different values ​​of m, and take the value of m when the compression advantage is the largest as the target value of m, denoted by M.

[0032] At this point, the target value of the number of virtual nodes m is obtained.

[0033] S003. Under the target value, a Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node, and the sequence to be compressed is compressed according to the code words corresponding to the virtual nodes and the real nodes in the Huffman tree to obtain a compression result.

[0034] In one embodiment, when the value of the number of virtual nodes m is the target value M, a Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node, and each virtual node and each real node respectively correspond to a node in the Huffman tree, then the codeword of the virtual node or the real node is the codeword of the corresponding node in the Huffman tree.

[0035] Each character in the sequence to be compressed is treated as a character to be encoded, and the characters to be encoded are compressed in sequence, specifically: When the character to be encoded is the first element in the sequence to be compressed, the codeword of the real node corresponding to the character to be encoded is used as the encoding result of the character to be encoded; When the character to be encoded is not the first element in the sequence to be compressed, determine whether the character to be encoded is the i-th associated character of the previous character, where i takes every integer in [1,M], and M represents the target value of the number of virtual nodes m. When the character to be encoded is not the i-th (i takes every integer in [1,M]) associated character of the previous character, the codeword of the real node corresponding to the character to be encoded is used as the encoding result of the character to be encoded; when the character to be encoded is the i-th associated character of the previous character, the codeword of the i-th virtual node is used as the encoding result of the character to be encoded.

[0036] The encoding results of all the characters to be encoded are concatenated together in the order of the characters to be encoded to obtain a compression result.

[0037] For example, when m=2, the compression advantage of the sequence to be compressed {a,b,a,b,a,a,b,a,c,b,a,c,d} is the greatest. The compression result obtained by compressing {a,b,a,b,a,a,b,a,c,b,a,c,d} using the method of the present invention is 001111001101110101. The result obtained by compressing {a,b,a,b,a,a,b,a,c,b,a,c,d} using traditional Huffman coding is 01101100110101110101100. The compression result corresponding to the present invention is 18 bits, and the compression result corresponding to the traditional Huffman coding is 23 bits. Compared with Huffman coding, the present invention improves the compression efficiency.

[0038] S004. Transmit the compression result.

[0039] The compression result is transmitted to the judicial data cloud server. In order to facilitate decoding, the Huffman tree and the i-th (i is every integer in [1,M]) associated character of each character need to be transmitted together with the compression result. Among them, M represents the target value of the number of virtual nodes m.

[0040] The codewords in the compression result are obtained by using the decoding method of Huffman coding, and the codewords in the compression result are decompressed in sequence, specifically: When the node corresponding to the codeword in the Huffman tree is a real node, the character corresponding to the real node is used as the decompression result of the codeword; when the node corresponding to the codeword in the Huffman tree is a virtual node, the serial number of the virtual node is recorded as i, and the value range of i is [1, M]. The i-th associated character of the decompression result of the previous codeword is used as the decompression result of the current codeword.

[0041] A sequence formed by decompression results of all codewords in the compression result is used as a decompression sequence.

[0042] The decompressed sequence is decoded using Base64 to obtain a binary sequence, and the binary sequence is decoded using the first encoding algorithm in step S001 to obtain judicial data.

[0043] Through the above steps, the compression, transmission and decompression of judicial data are completed.

[0044] An embodiment of the present invention also discloses a judicial data processing system based on big data, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the judicial data processing method based on big data according to the present invention is implemented.

[0045] The system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A judicial data processing method based on big data, characterized in that: include: Collect judicial data and convert the judicial data into sequences to be compressed; Obtain multiple associated characters for each character in the sequence to be compressed; Treat each character as a real node; Set m virtual nodes, where m corresponds to multiple different values; Under each value of m, the predicted frequency of each virtual node and the predicted frequency of each real node are determined according to the number of associated characters of each character and the number of associated characters of each character; the compression advantage is obtained according to the predicted frequency of each virtual node and the predicted frequency of each real node; The value of m when the compression advantage is the largest is taken as the target value; Under the target value, a Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node; the sequence to be compressed is compressed according to the code words corresponding to the virtual nodes and the real nodes in the Huffman tree to obtain a compression result; The compressed result is transmitted.

2. The judicial data processing method based on big data according to claim 1 is characterized in that: The step of obtaining multiple associated characters for each character in the sequence to be compressed includes: Any character in the sequence to be compressed is used as the target character. When a character in the sequence to be compressed appears at the next position of the target character, this character is said to be associated with the target character. The number of each character associated with the target character when it is associated with the target character is counted. Each character associated with the target character is sorted in descending order of the number of each character associated with the target character. Each sorted character is sequentially used as the first associated character, the second associated character, the third associated character, ..., the eth associated character of the target character, where e represents the number of types of characters associated with the target character.

3. The judicial data processing method based on big data according to claim 1 is characterized in that: The m corresponds to multiple different values, including: m traverses and takes [1, ], where S represents the number of character types in the sequence to be compressed.

4. The judicial data processing method based on big data according to claim 2 is characterized in that: The predicted frequency of the virtual node satisfies the expression: ； in, represents the predicted frequency of the i-th virtual node, i represents the sequence number of the virtual node, i takes every integer in [1,m], and m represents the number of virtual nodes; Indicates the length of the sequence to be compressed; Indicates the number of occurrences of the i-th associated character of the k-th character in the sequence to be compressed, where k represents the sequence number of the character. When the i-th associated character of the k-th character does not exist, ; S represents the number of character types in the sequence to be compressed.

5. The judicial data processing method based on big data according to claim 2 is characterized in that: The predicted frequency of the real node satisfies the expression: ； in, represents the predicted frequency of the jth real node, j represents the sequence number of the real node, j takes every integer in [1, S], and S represents the number of character types in the sequence to be compressed; Indicates the frequency of the character corresponding to the jth real node in the sequence to be compressed; Indicates the length of the sequence to be compressed; represents a function of the i-th associated character of the k-th character and the character corresponding to the j-th real node. When the character corresponding to the j-th real node is different from the i-th associated character of the k-th character, is 0, when the character corresponding to the jth real node is the same as the i-th associated character of the kth character, is the number of occurrences of the i-th associated character of the k-th character associated with the k-th character; when the i-th associated character of the k-th character does not exist, is 0.

6. The judicial data processing method based on big data according to claim 1 is characterized in that: The compression advantage satisfies the expression: ； in, Indicates compression advantage; represents the predicted frequency of the i-th virtual node, i represents the sequence number of the virtual node; m represents the number of virtual nodes; represents the predicted frequency of the jth real node, and j represents the sequence number of the real node; Represents the number of character types in the sequence to be compressed; exp() represents an exponential function with the natural constant e as the base.

7. The judicial data processing method based on big data according to claim 1 is characterized in that: The Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node, including: A Huffman tree is constructed according to the predicted frequency of each virtual node and the predicted frequency of each real node, and each virtual node and each real node respectively corresponds to a node in the Huffman tree.

8. The judicial data processing method based on big data according to claim 7 is characterized in that: The step of compressing the sequence to be compressed according to the code words corresponding to the virtual nodes and the real nodes in the Huffman tree to obtain the compression result includes: Each character in the sequence to be compressed is treated as a character to be encoded, and the characters to be encoded are compressed in sequence, specifically: When the character to be encoded is the first element in the sequence to be compressed, the codeword corresponding to the real node corresponding to the character to be encoded in the Huffman tree is used as the encoding result of the character to be encoded; When the character to be encoded is not the first element in the sequence to be compressed, determine whether the character to be encoded is the i-th associated character of the previous character, where i takes every integer in [1,M] and M represents the target value of m; when the character to be encoded is not the i-th associated character of the previous character, the code word corresponding to the real node corresponding to the character to be encoded in the Huffman tree is used as the encoding result of the character to be encoded; when the character to be encoded is the i-th associated character of the previous character, the code word corresponding to the i-th virtual node in the Huffman tree is used as the encoding result of the character to be encoded; The encoding results of all the characters to be encoded are concatenated together in the order of the characters to be encoded to obtain a compression result.

9. The judicial data processing method based on big data according to claim 1 is characterized in that: The step of converting the judicial data into a sequence to be compressed comprises: The judicial data is encoded into a binary sequence, and the binary sequence is encoded using Base64 to obtain a sequence to be compressed.

10. Judicial data processing system based on big data, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the judicial data processing method based on big data according to any one of claims 1 to 9 is implemented.

Images