A JSON-based internet of things data compression method

Abstract

This invention discloses a JSON-based IoT data compression method, belonging to the field of data compression. The method includes: loading a compression configuration file; reading in JSON text and obtaining the corresponding JSON object through deserialization; classifying the data using a Naive Bayes classifier; dividing the JSON object into template objects and data objects based on content update frequency; determining whether to update the template object; the compression party performing LZSS dictionary compression on the data object and sending it to the decompression party; and the decompression party decompressing and integrating the received compressed data to obtain the original data. This invention can compress JSON-formatted IoT data to a greater extent and has good adaptability to data belonging to different projects.

Description

Technical Field

[0001] This invention relates to the field of data compression, and more specifically, to a lossless compression method for IoT data based on JSON. Background Technology

[0002] In the field of the Internet of Things (IoT), the data collected by various sensing terminals is heterogeneous. To shield these differences and facilitate data parsing and extraction by the server when providing data to upper-layer applications, JSON format is used as the data transmission format between IoT terminals and remote cloud platforms. JSON is easy for humans to read and write, and easy for machines to generate and parse; however, it has a high degree of data redundancy. When the amount of data transmitted is too large, it leads to increased consumption of traffic and bandwidth resources, and increased transmission time.

[0003] Existing dedicated JSON data compression algorithms primarily achieve compression by altering the data structure, extracting the original JSON data into Templates and Values, and eliminating duplicate "Keys." However, when the JSON data has many attributes and few duplicate "Keys," the compression effect is not ideal. Conventional compression methods, such as Zip, can also compress JSON data, but they are not suitable for applications with limited processing capabilities, such as IoT terminals. Neither the dedicated nor general-purpose compression methods mentioned above consider the time-related redundancy issues between adjacent JSON data packets in IoT applications, leaving room for improvement in data compression efficiency. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a JSON-based IoT data compression method to achieve time and space redundancy removal of JSON data, and solve the problems of poor timeliness and high network resource consumption of large-volume JSON data during transmission.

[0005] The technical solution to achieve the purpose of this invention is as follows:

[0006] A JSON-based method for IoT data compression includes the following steps:

[0007] Step 1: Load the compressed configuration file;

[0008] Step 2: Read in a single IoT JSON data entry. Before reading, check the integrity of the data, and then deserialize it to obtain a JSON object.

[0009] Step 3: Use the Naive Bayes classifier to classify the JSON object: calculate the probability of each category appearing given the item to be classified, and select the class with the highest probability as the class label of the item to be classified.

[0010] Step 4: Divide the JSON object into a template object and a data object and store them in memory;

[0011] The template object includes the key value, the connector, and the identifier representing the value type from the source JSON object;

[0012] The data object includes only the Value from the source JSON object. For string Values, the double quotes are omitted, and adjacent Values ​​are separated by commas.

[0013] Step 5: Determine if a template object already exists locally and whether the content of the existing template object is consistent with the newly divided template object in Step 4; if yes, proceed to Step 6; otherwise, update the template objects of the compression and decompression sides to the newly divided template objects in Step 4 before proceeding to Step 6.

[0014] Step 6: Perform LZSS compression on the data objects divided in Step 4, and send the generated binary compressed data to the decompression party;

[0015] Step 7: The decompression party performs LZSS decompression on the received binary compressed data;

[0016] Step 8: Fill each decompressed Value into the template object in sequence to obtain the source JSON data. The data type of the Value can be deduced from the mapping relationship between the identifier and the data type in the template object.

[0017] The significant advantages of this invention compared to existing technologies are:

[0018] (1) This invention introduces a data classification mechanism into the data compression process, which overcomes the shortcomings of the traditional LZSS dictionary compression method, which cannot change the size of the sliding window according to the compressed content. This allows the method to maintain a high compression ratio for JSON data belonging to different projects.

[0019] (2) This invention takes into account the time-related redundancy problem in adjacent JSON data messages in the Internet of Things environment. It utilizes the characteristic that different parts of the JSON data are updated at different frequencies over time to divide the source data into data objects and template objects. By maintaining a relatively stable template object on the compression and decompression sides, the improved LZSS dictionary compression and transmission are performed only on the frequently updated data objects, thereby reducing the total amount of data compression and transmission, further improving the compression rate, and reducing the amount of computation. Attached Figure Description

[0020] Figure 1 This is a diagram of the data compression and transmission model of the present invention;

[0021] Figure 2 This is a diagram illustrating the execution steps of the Naive Bayes classifier of this invention.

[0022] Figure 3 This is a schematic diagram of the LZSS compression encoding process of the present invention;

[0023] Figure 4 This is a schematic diagram of the IoT JSON data partitioning process of the present invention; Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0025] Combination Figure 1 The present invention provides a JSON-based IoT data compression method, comprising the following steps:

[0026] Step 1: Load the compression configuration file. This configuration file describes the optimal sliding window for different types of JSON data. Its value can be obtained through actual testing. That is, the same type of data is compressed using LZSS compression algorithms with different window sizes, and the sliding window that first achieves a high compression ratio is taken as the optimal sliding window.

[0027] Step 2: Read in a single IoT JSON data entry. Before reading, check the integrity of the data, and then deserialize it to obtain a JSON object.

[0028] Step 3: Classify the JSON object using a Naive Bayes classifier. The Naive Bayes classifier is a classification method based on Bayes' theorem and the assumption of conditional independence of features. Its basic idea is to calculate the probability of each category occurring given the item to be classified, and select the class with the highest probability as the class label for the item. The specific mathematical model is as follows:

[0029]

[0030] Where y is the classification category, C is the set of possible values ​​for y, P(y) represents the probability of a certain category occurring, n is the number of features, and x i Let x be the value of the item to be classified on the i-th feature, and P(x) be the value of the item to be classified on the i-th feature. i |y) represents the conditional probability estimate of each feature attribute of the item to be classified under a certain category, y Naive This represents the mapping relationship between the item to be classified and the classification category, that is, the most likely category y for a given item x in the Bayesian classifier.

[0031] The basic process of classifying data using a Naive Bayes classifier is as follows: Figure 2 As shown, the specific implementation steps are as follows:

[0032] (1) Select the features of the data (keywords are used as data features in this embodiment), divide each feature attribute, and then manually classify a part of the items to be classified to form a training set;

[0033] (2) Calculate the probability P(y) of each category appearing in the training set and the conditional probability P(x) of each feature attribute under each category. i |y), and record the result;

[0034] (3) Using the results obtained in step (2), calculate the product of the classification probability and the feature probability given the item to be classified, i.e., calculate... The value of is used to select the class y that maximizes the product as the class label for the item to be classified.

[0035] Step 4: Divide the JSON object into template objects and data objects and store them in memory. The basis for dividing the source JSON object is that different parts of the JSON data update at different frequencies over time. For example, keys update much less frequently than values. If the less frequently updated content in the source JSON is extracted as template objects, and these template objects are maintained on both the decompression and compression sides, and only the frequently updated data objects are compressed and transmitted, the overall data compression and transmission volume can be reduced, improving compression efficiency. The process of dividing the source JSON object into data objects is as follows: Figure 3 As shown;

[0036] The template object includes the key value, connector, and identifier representing the value type in the source JSON object. The mapping relationship between the JSON data type and the identifier used in this embodiment is as follows: number - 'N', string - 'S', logical value - 'L', null value - 'E'.

[0037] The data object includes only the Value from the source JSON object. For string Values, the double quotes are omitted, and adjacent Values ​​are separated by commas.

[0038] Step 5: Determine if a template object already exists locally and whether its content is consistent with the newly defined template object in Step 4. If yes, proceed to Step 6; otherwise, update the template objects of both the compression and decompression sides to the newly defined template object in Step 4 before proceeding to Step 6.

[0039] Step 6: Perform LZSS compression on the data objects divided in Step 4, and send the generated binary compressed data to the decompression party;

[0040] The encoding process of LZSS dictionary compression is as follows: Figure 4As shown, this compression method uses a virtual, sliding window as a dictionary, replacing the actual string with the position and length of the current string in the dictionary to reduce file size. However, the size of the sliding window used in this method is fixed. In practical IoT applications, data from different projects often differ significantly. If the sliding window is too large, the search computation increases, leading to longer compression times; if the sliding window is too small, it becomes difficult to find a matching string in the dictionary, resulting in a low compression ratio. This invention utilizes the classification results from step 3 to find the optimal sliding window matching the data type in the configuration file, ensuring a consistently high compression ratio for JSON data belonging to different projects.

[0041] Step 7: The decompression party performs LZSS decompression on the received binary compressed data;

[0042] Step 8: Fill each decompressed Value into the template object in sequence to obtain the source JSON data. The data type of the Value can be deduced from the mapping relationship between the identifier and the data type in the template object.

[0043] The technical effects of the method in the embodiments of the present invention are verified as follows:

[0044] Table 1 Comparison of Compression Effects

[0045] Test samples JSON1 JSON2 JSON3 JSON4 Original file (unit: KB) 49.79 99.17 149.92 199.93 CJSON 28.39 56.35 84.22 111.9 Zip 12.91 25.23 37.19 49.25 This invention 7.61 15.09 21.3 27.76

[0046] Table 1 above compares the compression effects of the present invention with those of two existing compression algorithms. It can be seen that the present invention has a higher compression ratio for JSON data compared to the dedicated compression algorithm CJSON and the general compression algorithm ZIP.

Claims

1. A JSON-based IoT data compression method, characterized in that, Includes the following steps: Step 1: Load the compressed configuration file; Step 2: Read in a single IoT JSON data entry. Before reading, check the integrity of the data, and then deserialize it to obtain a JSON object. Step 3: Use the Naive Bayes classifier to classify the JSON object: calculate the probability of each category appearing given the item to be classified, and select the class with the highest probability as the class label of the item to be classified. Step 4: Divide the JSON object into a template object and a data object and store them in memory; The template object includes the key value, the connector, and the identifier representing the value type from the source JSON object; The data object includes only the Value from the source JSON object. For string Values, the double quotes are omitted, and adjacent Values ​​are separated by commas. Step 5: Determine if a template object already exists locally and whether the content of the existing template object is consistent with the newly divided template object in Step 4; if yes, proceed to Step 6; otherwise, update the template objects of the compression and decompression sides to the newly divided template objects in Step 4 before proceeding to Step 6. Step 6: Perform LZSS compression on the data objects divided in Step 4, and send the generated binary compressed data to the decompression party; Step 7: The decompression party performs LZSS decompression on the received binary compressed data; Step 8: Fill the decompressed Values ​​into the template object in sequence to obtain the source JSON data. The data type of the Value is determined by the mapping relationship between the identifier and the data type in the template object.

2. The JSON-based IoT data compression method according to claim 1, characterized in that, The configuration file describes the optimal sliding window for different types of JSON data. Using LZSS compression, the optimal sliding window that matches the corresponding data type is found in the configuration file based on the classification results in step 3.

3. The JSON-based IoT data compression method according to claim 1, characterized in that, The specific steps for classifying data using the Naive Bayes classifier are as follows: (1) Select the features of the data, divide each feature attribute, and then manually classify a part of the items to be classified to form a training set; (2) Calculate the probability of each category appearing in the training set and the conditional probability of each feature attribute under each category, and record the results; (3) Using the results obtained in step (2), calculate the product of the classification probability and the feature probability given the item to be classified, and take the class that maximizes the product as the class label of the item to be classified.

4. The JSON-based IoT data compression method according to claim 1, characterized in that, The mathematical model of the Naive Bayes classifier is as follows: in It is a category. yes The set of possible values, This represents the probability of a certain category occurring, where n is the number of features. Items to be classified In the The values ​​of each feature This is a conditional probability estimate for each feature attribute of an item to be classified within a certain category. This represents the mapping relationship between the item to be classified and the classification category.

Images