Data management method and system of Internet of Things equipment based on big data

Abstract

The invention relates to the field of data processing, in particular to a data management method and system for Internet of Things equipment based on big data. The method comprises the following steps: acquiring an Internet of Things equipment data sequence; obtaining the occurrence probability of each value in the Internet of Things equipment data sequence, recording the occurrence probability as a comprehensive probability, and obtaining the distribution preposition degree of any value; calculating an adjustment coefficient of each value, wherein the adjustment coefficient is in positive correlation with the distribution preposition degree; setting a segmentation proportion of each value according to the adjustment coefficient and the comprehensive probability, wherein the segmentation proportion is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability; and coding control is carried out according to the segmentation proportion to realize data storage management. The segmentation proportion is adjusted according to the data arrangement information, and the compression effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data management method and system for an Internet of Things device based on big data. Background Art

[0002] With the rapid development of IoT technology, IoT devices have been widely used in various fields. For example, smart homes need to collect environmental information in real time for better control and perform real-time control based on the environmental information. During this process, a large amount of IoT device data will be generated. In order to facilitate subsequent management and control, these IoT device data need to be stored. As the amount of IoT device data accumulates over time, a large amount of storage space will be required. In order to save storage costs, IoT device data needs to be compressed and stored.

[0003] Arithmetic coding algorithm is a commonly used coding compression algorithm. This method selects the intervals corresponding to the data to be encoded in turn according to the arrangement of the data to be encoded, and performs hierarchical interval division until all the data to be encoded participate in the interval division, and selects one data in the interval as the coding data. The length of the coded data obtained by the coding algorithm is not only related to the data frequency, but also has a great relationship with the arrangement of the data to be encoded. Among them, for the same data set, the coding length obtained under some arrangements is longer, and the coding length obtained under other arrangements is shorter, so the traditional arithmetic coding algorithm has a poor compression effect on the data under some arrangements. Therefore, how to reduce the larger coding length caused by the arrangement becomes the research focus of the present invention.

[0004] The patent document with authorization announcement number CN118233499B discloses a data management method and system based on the Internet of Things. The method in the patent document mainly involves content to assist decision-making through data analysis. The method in the patent document does not involve content of coding compression. Therefore, the method in the patent document cannot solve the technical problems of this solution. Summary of the invention

[0005] In order to solve the problem of how to reduce the larger code length caused by the arrangement, the present invention provides a data management method and system for an Internet of Things device based on big data.

[0006] In a first aspect, the present invention provides a data management method for IoT devices based on big data, which adopts the following technical solution: A data management method for IoT devices based on big data, comprising the steps of: Get IoT device data sequence; The probability of occurrence of each value in the IoT device data sequence is recorded as the comprehensive probability, and the distribution pre-proportion of any value is obtained. , use several sliding windows of different lengths to slide on the IoT device data sequence in turn, divide the data in each sliding window of each length into two segments, obtain the occurrence probability of the value in each segment and the distribution center of the value in each segment, It represents the distance between the centroids of the values ​​in the two segments obtained by the j-th sliding window of the i-th length. , They represent the probability of occurrence of the value in one segment and the probability of occurrence of the value in another segment obtained by the j-th sliding window of the i-th length, respectively. represents the number of sliding windows of the i-th length, Indicates the number of types of sliding window lengths; Calculate the adjustment coefficient of each value, the adjustment coefficient is positively correlated with the distribution front degree; set the split ratio of each value according to the adjustment coefficient and the comprehensive probability, the split ratio is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability; Encoding control is performed according to the segmentation ratio to achieve data storage management.

[0007] The present invention takes into account that the code length is shorter when the interval shrinkage speed is fast in the early stage of coding and slow in the late stage of coding, and the main factor affecting the interval shrinkage speed is the segmentation ratio, so the code length is reduced by adjusting the segmentation ratio; further, when adjusting the segmentation ratio, it is considered that if the data of a type of values ​​is mainly distributed on the front side of the sequence, the shrinkage speed of the early interval can be increased by reducing the segmentation ratio of the data of this type of values, and if the data of a type of values ​​is mainly distributed on the back side of the sequence, the shrinkage speed of the late interval can be reduced by increasing the segmentation ratio of the data of this type of values, thereby achieving accurate adjustment of the segmentation ratio; further, it is considered that the segmentation ratio is related to the position of the data of each type of values ​​in the sequence, and thus the position of the data of each type of values ​​in the sequence is accurately described by analyzing the distribution advance degree of the data of each type of values; further, when calculating the distribution advance degree, the position of the data of each type of values ​​in the sequence is accurately reflected by introducing the center of gravity of each type of values ​​and the distance between the centers of gravity.

[0008] Preferably, the method of using several sliding windows of different lengths to slide on the IoT device data sequence in sequence includes: Preset sliding windows of several lengths; First, align the left end of the IoT device data sequence with the left end of a sliding window of any length, and then use the sliding window of any length to slide on the IoT device data sequence with 1 as the sliding step to obtain several sliding windows of the length.

[0009] The present invention measures the distribution of data in a sequence at different scales by setting sliding windows of different lengths, thereby improving the accuracy of data position measurement.

[0010] Preferably, the method for obtaining the distribution center of gravity of the value in each segment includes: The position of each data of this type of value in any segment is obtained, and the geometric center of all data of this type of value is obtained according to the position of all data of this type of value in the segment and recorded as the distribution center of gravity of this type of value in the segment.

[0011] The present invention reflects the center of gravity of the data through the geometric center of the position of the data of this value. This measurement method is relatively simple and has high implementation efficiency.

[0012] Preferably, the calculation of the adjustment coefficient for each value includes: The distribution front degree of each value is normalized to obtain the adjustment coefficient of each value.

[0013] The present invention adjusts the distribution pre-position degree to the same dimension through normalization processing, laying a foundation for the subsequent accurate adjustment of the segmentation ratio.

[0014] Preferably, the step of setting the split ratio of each value according to the adjustment coefficient and the comprehensive probability includes: The comprehensive probability of each value is divided by the adjustment coefficient and then normalized to obtain the segmentation ratio of each value.

[0015] The present invention reduces the segmentation ratio of data at the front end of the sequence by dividing by an adjustment coefficient, thereby increasing the shrinkage speed of the early coding interval, and increases the segmentation ratio of data at the back end of the sequence, thereby reducing the shrinkage speed of the late coding interval, thereby achieving the effect of reducing the coding length.

[0016] Preferably, the encoding control according to the segmentation ratio includes: Arrange the segmentation ratios of all kinds of values ​​in descending order, record the interval from 0 to 1 as the first-level interval, and divide the first-level interval into K intervals as the second-level interval according to the segmentation ratios and arrangement order of all kinds of values, where K represents the number of types of values; obtain the second-level interval of the value corresponding to the first data in the IoT device data sequence as the second-level target interval, and divide the second-level target interval into K intervals as the third-level interval according to the segmentation ratios and arrangement order of all kinds of values; and so on, until the interval division is completed according to the second-to-last data in the IoT device data sequence to obtain the final-level interval, and randomly select a data in the final-level target interval of the value corresponding to the last data in the IoT device data sequence as the encoding sequence.

[0017] Preferably, the method for implementing data storage management includes: Stores the encoded sequence as a storage object.

[0018] Preferably, the occurrence probability of each value in the IoT device data sequence is recorded as a comprehensive probability, including: The number of occurrences of each value in the IoT device data sequence is obtained, and the number of occurrences of each value is divided by the number of IoT device data sequences to obtain the probability of occurrence of each value, which is recorded as the comprehensive probability of each value.

[0019] Preferably, the data in each sliding window of each length is divided into two segments in a uniform segmentation manner.

[0020] In a second aspect, the present invention provides a data management system for IoT devices based on big data, which adopts the following technical solution: A data management system for Internet of Things devices based on big data includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned data management method for Internet of Things devices based on big data is implemented.

[0021] By adopting the above technical solution, the above-mentioned data management method for IoT devices based on big data is generated into a computer program and stored in a memory to be loaded and executed by a processor, so that a terminal device is made according to the memory and the processor for easy use.

[0022] The present invention has the following technical effects: The present invention takes into account that the interval reduction speed is fast in the early stage of coding and slow in the late stage of coding, and the code length is shorter, and the main factor affecting the interval reduction speed is the segmentation ratio, so the code length is reduced by adjusting the segmentation ratio; Furthermore, when adjusting the split ratio, if the data of a type of values ​​is mainly distributed at the front side of the sequence, the speed of shrinking the early interval can be increased by reducing the split ratio of the data of this type of values, and if the data of a type of values ​​is mainly distributed at the back side of the sequence, the speed of shrinking the late interval can be reduced by increasing the split ratio of the data of this type of values, thereby achieving accurate split ratio adjustment; Furthermore, considering that the split ratio is related to the position of the data of each value in the sequence, the position of the data of each value in the sequence can be accurately described by analyzing the distribution advance degree of the data of each value; Furthermore, when calculating the distribution front degree, the centroid of each type of value and the distance between the centroids are introduced to accurately reflect the position of the data of each type of value in the sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0024] Figure 1 It is a flow chart of a method in a data management method of an Internet of Things device based on big data in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] It should be understood that when the terms "first", "second", etc. are used in the claims, descriptions, and drawings of the present invention, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0027] The embodiment of the present invention discloses a data management method for an IoT device based on big data, referring to Figure 1 , comprising steps S1 to S5: S1: Get IoT device data sequence.

[0028] Specifically, the IoT device data at each moment is obtained, and the IoT device data at all moments are arranged in time sequence to obtain an IoT device data sequence. In this embodiment, the control data of the smart home is used as the IoT device data, and other embodiments may use other data as the IoT device data, which is not specifically limited in this embodiment.

[0029] S2: Obtain the occurrence probability of each value in the IoT device data sequence and record it as the comprehensive probability, and obtain the distribution leading degree of any value.

[0030] It should be noted that the encoding value obtained by encoding the sequence AAAAAAAAABBBBCCD using the arithmetic coding algorithm is: 0.0042322, and the encoding result obtained by encoding the sequence DCCBBBBAAAAAAAAA is: 0.99536; it can be seen from the observation that the two sequences contain the same type of data, all of which are A, B, C and D. The frequency of occurrence of each data is also the same, and the only difference is the different arrangement. However, the encoding length caused by the different arrangement is 2. This is only a sequence of length 16, and its encoding length can differ by 2. If the sequence is longer, the difference in encoding length is greater. Therefore, in order to improve the compression effect, it is necessary to reduce the problem of increased encoding length caused by the arrangement. Through analysis, it can be seen that the biggest difference between the two sequences is the difference in the arrangement of data of different frequencies. The first sequence arranges high-frequency data in front and low-frequency data in the back; the second sequence arranges low-frequency data in front and high-frequency data in the back. When high-frequency data is arranged in front, the shrinking speed of the early interval is slow, and the shrinking speed of the later interval is fast, which will shrink the final interval to a smaller size; when low-frequency data is arranged in front, the shrinking speed of the early interval is fast, and the shrinking speed of the later interval is slow, which will not shrink the final interval to a smaller size. In order to reduce the code growth problem caused by the arrangement method, it is necessary to increase the shrinking speed of the early interval and reduce the shrinking speed of the later interval.

[0031] S20: Obtain the occurrence probability of each value in the IoT device data sequence and record it as the comprehensive probability.

[0032] Preferably, as an example, the occurrence probability of each value in the IoT device data sequence is obtained and recorded as a comprehensive probability, including: The number of occurrences of each value in the IoT device data sequence is obtained, and the number of occurrences of each value is divided by the number of IoT device data sequences to obtain the probability of occurrence of each value, which is recorded as the comprehensive probability of each value.

[0033] S21: Obtain the distribution leading degree of any value.

[0034] It should be noted that the segmentation ratio determines the speed at which the interval is reduced. In order to increase the reduction speed of the early interval and reduce the reduction speed of the later interval, the segmentation ratio of the data mainly distributed in the left segment needs to be lowered, and the segmentation ratio of the data mainly distributed in the right segment needs to be increased.

[0035] It should be further explained that in order to adjust the segmentation ratio, the distribution advance degree of the data of each value needs to be analyzed, and the distribution advance degree of the data of each value reflects the distribution of the data of each value in each segment.

[0036] Preferably, as an example, obtaining the distribution leading degree of any value includes:

[0037] Among them, several sliding windows of different lengths are used to slide on the IoT device data sequence in turn, and the data in each sliding window of each length is divided into two segments, and the occurrence probability of the value in each segment and the distribution center of the value in each segment are obtained. It represents the distance between the centroids of the values ​​in the two segments obtained by the j-th sliding window of the i-th length. , They represent the probability of occurrence of the value in one segment and the probability of occurrence of the value in another segment obtained by the j-th sliding window of the i-th length, respectively. represents the number of sliding windows of the i-th length, Indicates the number of types of sliding window lengths.

[0038] Understandably, It reflects the probability difference between the two segments. The larger the value, the more data of this value is distributed in the left segment, so the distribution of data of this value is more advanced. It reflects the position of the data of this value in the segment. The larger the value, the greater the degree to which the data of this value is at the front or back end of the sequence. When the value is larger, the data is not only distributed in the left segment, but also distributed to a greater extent on the left side of the left segment. Therefore, the distribution of data with this value has a greater front-end degree.

[0039] The above embodiments set the sliding window of each length, the segment, the probability of occurrence of this type of value in the segment, and the distribution center of this type of value in the segment. The following describes the method for obtaining the sliding window of each length, the segment, the probability of occurrence of this type of value in the segment, and the distribution center of this type of value in the segment.

[0040] First, the method for obtaining sliding windows of each length includes: Preset sliding windows of several lengths; First, align the left end of the IoT device data sequence with the left end of a sliding window of any length, and then slide the sliding window of that length on the IoT device data sequence with 1 as the sliding step until the right end of the sliding window is aligned with the right end of the IoT device data sequence, and obtain several sliding windows of that length. Integers between , other embodiments may take other values ​​for the length of the sliding window, and this embodiment does not impose any specific limitation.

[0041] Then, the segmented acquisition method includes: Each sliding window of each length is evenly divided into two segments to obtain two segments.

[0042] Then, the method for obtaining the occurrence probability of the value in the segment includes: The number of occurrences of the data with this value in the segment is obtained, and the number of occurrences of the data with this value in the segment is divided by the number of data in the segment to obtain the probability of occurrence of the data with this value in the segment.

[0043] Finally, the method for obtaining the distribution center of this value in the segment includes: The position of each data of this type of value in any segment is obtained, and the geometric center of all data of this type of value is obtained according to the position of all data of this type of value in the segment and recorded as the distribution center of gravity of this type of value in the segment.

[0044] S3: Calculate the adjustment coefficient of each value, the adjustment coefficient is positively correlated with the distribution front degree; set the segmentation ratio of each value according to the adjustment coefficient and the comprehensive probability, the segmentation ratio is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability.

[0045] S30: Calculate the adjustment coefficient for each value.

[0046] Preferably, as an example, calculating the adjustment coefficient for each value includes: The distribution front degree of each value is normalized to obtain the adjustment coefficient of each value.

[0047] This embodiment adopts the maximum and minimum normalization method for normalization processing. Other embodiments may adopt other normalization methods, and this embodiment does not make any specific limitation.

[0048] S31: Setting the split ratio of each value according to the adjustment coefficient and the comprehensive probability.

[0049] It should be noted that the traditional arithmetic coding algorithm does not consider the data distribution, and directly uses the data probability as the segmentation ratio. In order to reduce the increase in code length caused by data arrangement, the segmentation ratio is adjusted using an adjustment coefficient.

[0050] It should be further explained that in order to make the shrinking speed of the early interval faster and the shrinking speed of the later interval slower, the splitting ratio of the values ​​with a large distribution front degree should be reduced, and the splitting ratio of the values ​​with a small distribution front degree should be increased.

[0051] Preferably, as an example, the split ratio of each value is set according to the adjustment coefficient and the comprehensive probability, including: The comprehensive probability of each value is divided by the adjustment coefficient and then normalized to obtain the segmentation ratio of each value.

[0052] S4: Encoding control is performed according to the segmentation ratio to achieve data storage management.

[0053] Preferably, as an example, encoding control is performed according to the segmentation ratio to implement data storage management, including: Arrange the segmentation ratios of all kinds of values ​​in descending order, record the interval from 0 to 1 as the first-level interval, and divide the first-level interval into K intervals as the second-level interval according to the segmentation ratios and arrangement order of all kinds of values, where K represents the number of types of values; obtain the second-level interval of the value corresponding to the first data in the IoT device data sequence as the second-level target interval, and divide the second-level target interval into K intervals as the third-level interval according to the segmentation ratios and arrangement order of all kinds of values; and so on, until the interval division is completed according to the second-to-last data in the IoT device data sequence to obtain the final-level interval, and randomly select a data in the final-level target interval of the value corresponding to the last data in the IoT device data sequence as the encoding sequence.

[0054] The coding sequence and the split ratio of each value are stored as storage objects.

[0055] S5: Decode the coded sequence.

[0056] It should be noted that decoding a coded sequence according to a segmentation ratio is a prior art and will not be described in detail here.

[0057] An embodiment of the present invention also discloses a data management system for Internet of Things devices based on big data, including a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, a data management method for Internet of Things devices based on big data according to the present invention is implemented.

[0058] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, and their configuration and functions are known in the art, so they will not be described in detail here.

[0059] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory, a dynamic random access memory, a static random access memory, an enhanced dynamic random access memory, a high bandwidth memory, a hybrid storage cube, etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device.

[0060] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

[0061] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data management method for IoT devices based on big data, characterized in that: Includes steps: Get IoT device data sequence; The probability of occurrence of each value in the IoT device data sequence is recorded as the comprehensive probability, and the distribution pre-proportion of any value is obtained. , use several sliding windows of different lengths to slide on the IoT device data sequence in turn, divide the data in each sliding window of each length into two segments, obtain the occurrence probability of the value in each segment and the distribution center of the value in each segment, It represents the distance between the centroids of the values ​​in the two segments obtained by the j-th sliding window of the i-th length. , They represent the probability of occurrence of the value in one segment and the probability of occurrence of the value in another segment obtained by the j-th sliding window of the i-th length, respectively. represents the number of sliding windows of the i-th length, Indicates the number of types of sliding window lengths; calculates the adjustment coefficient for each value, the adjustment coefficient is positively correlated with the distribution front degree; sets the segmentation ratio for each value according to the adjustment coefficient and the comprehensive probability, the segmentation ratio is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability; performs encoding control according to the segmentation ratio to realize data storage management.

2. The data management method of IoT devices based on big data according to claim 1, characterized in that: The method of using sliding windows of different lengths to slide on the IoT device data sequence in sequence includes: Preset sliding windows of several lengths; First, align the left end of the IoT device data sequence with the left end of a sliding window of any length, and then use the sliding window of any length to slide on the IoT device data sequence with 1 as the sliding step to obtain several sliding windows of the length.

3. The data management method of IoT devices based on big data according to claim 1, characterized in that: The method for obtaining the distribution center of gravity of the value in each segment includes: The position of each data of this type of value in any segment is obtained, and the geometric center of all data of this type of value is obtained according to the position of all data of this type of value in the segment and recorded as the distribution center of gravity of this type of value in the segment.

4. The data management method of IoT devices based on big data according to claim 1, characterized in that: The calculation of the adjustment coefficient for each value includes: The distribution front degree of each value is normalized to obtain the adjustment coefficient of each value.

5. The data management method of IoT devices based on big data according to claim 4 is characterized in that: The step of setting the split ratio of each value according to the adjustment coefficient and the comprehensive probability includes: The comprehensive probability of each value is divided by the adjustment coefficient and then normalized to obtain the segmentation ratio of each value.

6. The data management method of IoT devices based on big data according to claim 1, characterized in that: The encoding control according to the segmentation ratio includes: Arrange the segmentation ratios of all kinds of values ​​in descending order, record the interval from 0 to 1 as the first-level interval, and divide the first-level interval into K intervals as the second-level interval according to the segmentation ratios and arrangement order of all kinds of values, where K represents the number of types of values; obtain the second-level interval of the value corresponding to the first data in the IoT device data sequence as the second-level target interval, and divide the second-level target interval into K intervals as the third-level interval according to the segmentation ratios and arrangement order of all kinds of values; and so on, until the interval division is completed according to the second-to-last data in the IoT device data sequence to obtain the final-level interval, and randomly select a data in the final-level target interval of the value corresponding to the last data in the IoT device data sequence as the encoding sequence.

7. The data management method of IoT devices based on big data according to claim 6, characterized in that: The data storage management is implemented by: Stores the encoded sequence as a storage object.

8. The data management method of IoT devices based on big data according to claim 1, characterized in that: The occurrence probability of each value in the IoT device data sequence is recorded as the comprehensive probability, including: The number of occurrences of each value in the IoT device data sequence is obtained, and the number of occurrences of each value is divided by the number of IoT device data sequences to obtain the probability of occurrence of each value, which is recorded as the comprehensive probability of each value.

9. The data management method of IoT devices based on big data according to claim 1, characterized in that: The data in each sliding window of each length is divided into two segments in a uniform segmentation manner.

10. A data management system for IoT devices based on big data, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a data management method for an Internet of Things device based on big data is implemented according to any one of claims 1 to 9.

Images