A data management method and system for Internet of Things devices based on big data

Abstract

The present invention relates to the field of data processing, and particularly to a data management method and system for Internet of Things devices based on big data. The method includes the steps of: obtaining an Internet of Things device data sequence; obtaining the occurrence probability of each value in the Internet of Things device data sequence, which is recorded as the comprehensive probability, and obtaining the distribution leading degree of any value; calculating the adjustment coefficient of each value, where the adjustment coefficient is positively correlated with the distribution leading degree; setting the segmentation ratio of each value according to the adjustment coefficient and the comprehensive probability, where the segmentation ratio is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability; performing coding control according to the segmentation ratio to achieve data storage management. Adjust the segmentation ratio according to the data arrangement information to improve the compression effect.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a data management method and system for Internet of Things devices based on big data. Background Art

[0002] With the rapid development of Internet of Things technology, Internet of Things devices have been widely used in various fields. For example, in smart homes, in order to better control, environmental information needs to be collected in real time and real-time control is performed according to the environmental information. A large amount of Internet of Things device data will be generated during this process. In order to facilitate subsequent management and control, these Internet of Things device data need to be stored. As time accumulates, the amount of Internet of Things device data gradually increases, which will require a large amount of storage space. In order to save storage costs, it is necessary to compress and store the Internet of Things device data.

[0003] As a commonly used coding and compression algorithm, the arithmetic coding algorithm sequentially selects the intervals corresponding to the data to be encoded for hierarchical interval division according to the arrangement of the data to be encoded until all the data to be encoded participate in the interval division, and then selects a data in the interval as the encoded data. The length of the encoded data obtained by using this coding algorithm is not only related to the data frequency, but also has a great relationship with the arrangement of the data to be encoded. Among them, for the same data set, the encoding length obtained under some arrangements is longer, and the encoding length obtained under some other arrangements is shorter. Therefore, the traditional arithmetic coding algorithm has a poor compression effect on the data under some arrangements. Therefore, how to reduce the large coding length caused by the arrangement has become the research focus of the present invention.

[0004] The patent document with the authorization announcement number CN118233499B discloses a data management method and system based on the Internet of Things. The method in this patent document mainly involves the content of assisting decision-making through data analysis, and the method in this patent document does not involve the content of coding and compression. Therefore, the technical problems of the present solution cannot be solved by using the method in this patent document. Summary of the Invention

[0005] In order to solve the problem of how to reduce the large coding length caused by the arrangement, the present invention provides a data management method and system for Internet of Things devices based on big data.

[0006] In the first aspect, the present invention provides a data management method for Internet of Things devices based on big data, adopting the following technical solutions:

[0007] A data management method for Internet of Things devices based on big data includes the steps of:

[0008] Obtain an Internet of Things device data sequence;

[0009] Obtain the occurrence probability of each value in the data sequence of the Internet of Things device, which is recorded as the comprehensive probability, and obtain the distribution preposition degree of any value. Use sliding windows of several different lengths to slide on the data sequence of the Internet of Things device in turn. Divide the data in each sliding window of each length into two segments, obtain the occurrence probability of this value in each segment and the distribution center of gravity of this value in each segment. Represents the distance between the centers of gravity of this value in the two segments obtained from the j-th sliding window of the i-th length. 、 Respectively represent the occurrence probability of this value in one segment and the occurrence probability of this value in the other segment obtained from the j-th sliding window of the i-th length. Represents the number of sliding windows of the i-th length. Represents the number of types of sliding window lengths;

[0010] Calculate the adjustment coefficient of each value, and the adjustment coefficient is positively correlated with the distribution preposition degree; set the segmentation ratio of each value according to the adjustment coefficient and the comprehensive probability, and the segmentation ratio is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability;

[0011] Perform coding control according to the segmentation ratio to achieve data storage management.

[0012] The present invention takes into account that when the interval reduction speed is fast in the early stage of coding and slow in the later stage of coding, the obtained coding length is shorter, and the main factor affecting the interval reduction speed is the segmentation ratio. Therefore, the coding length is reduced by adjusting the segmentation ratio; further, when adjusting the segmentation ratio, considering that if the data of a certain type of value is mainly distributed on the front side of the sequence, the segmentation ratio of the data of this type of value can be reduced to increase the interval reduction speed in the early stage. If the data of a certain type of value is mainly distributed on the rear side of the sequence, the segmentation ratio of the data of this type of value can be increased to reduce the interval reduction speed in the later stage, so as to achieve accurate adjustment of the segmentation ratio; further, considering that the segmentation ratio is related to the position of the data of each type of value in the sequence, the distribution preposition degree of the data of each type of value is analyzed to accurately describe the position of the data of each type of value in the sequence; further, when calculating the distribution preposition degree, the center of gravity of each type of value and the distance between the centers of gravity are introduced to accurately reflect the position of the data of each type of value in the sequence.

[0013] Preferably, the use of sliding windows of several different lengths to slide on the data sequence of the Internet of Things device in turn includes:

[0014] Preset sliding windows of several lengths;

[0015] First, align the left end of the IoT device data sequence with the left end of a sliding window of any length. Then, slide the sliding window of this length on the IoT device data sequence with a sliding step of 1 to obtain several sliding windows of this length.

[0016] The present invention measures the distribution of data in the sequence at different scales by setting sliding windows of different lengths, thereby improving the accuracy of data position measurement.

[0017] Preferably, the method for obtaining the distribution centroid of this value in each segment includes:

[0018] Obtain the positions of the data of this value in any segment, and obtain the geometric center of all the data of this value in this segment according to the positions of all the data of this value in this segment, which is recorded as the distribution centroid of this value in this segment.

[0019] The present invention reflects the centroid of the data through the geometric center of the positions of the data of this value, and this measurement method is relatively simple and has high implementation efficiency.

[0020] Preferably, the calculation of the adjustment coefficient for each value includes:

[0021] Normalize the degree of distribution front position of each value to obtain the adjustment coefficient for each value.

[0022] The present invention adjusts the degree of distribution front position to the same dimension through normalization, laying a foundation for accurately adjusting the segmentation ratio subsequently.

[0023] Preferably, the setting of the segmentation ratio for each value according to the adjustment coefficient and the comprehensive probability includes:

[0024] Normalize the result of dividing the comprehensive probability of each value by the adjustment coefficient to obtain the segmentation ratio for each value.

[0025] The present invention reduces the segmentation ratio of the data at the front end of the sequence by dividing by the adjustment coefficient, thereby increasing the reduction speed of the interval in the early stage of coding, and increases the segmentation ratio of the data at the back end of the sequence, thereby reducing the reduction speed of the interval in the later stage of coding, so as to achieve the effect of reducing the coding length.

[0026] Preferably, the coding control according to the segmentation ratio includes:

[0027] Arrange the segmentation ratios of all kinds of values in descending order. Denote the interval from 0 to 1 as the first-layer interval. According to the segmentation ratios of all kinds of values and the arrangement order, divide the first-layer interval into K intervals, which are denoted as the second-layer intervals, where K represents the number of value types. Obtain the second-layer interval corresponding to the value of the first data in the IoT device data sequence, which is denoted as the second-layer target interval. According to the segmentation ratios of all kinds of values and the arrangement order, divide the second-layer target interval into K intervals to obtain the third-layer intervals. And so on, until the final-layer intervals are obtained according to the penultimate data in the IoT device data sequence. Randomly select a data from the final-layer target interval corresponding to the value of the last data in the IoT device data sequence as the coding sequence.

[0028] Preferably, the implementation of data storage management includes:

[0029] Store the coding sequence as a storage object.

[0030] Preferably, obtaining the occurrence probability of each value in the IoT device data sequence, which is denoted as the comprehensive probability, includes:

[0031] Obtain the occurrence times of each value in the IoT device data sequence. Divide the occurrence times of each value by the number of the IoT device data sequence to obtain the occurrence probability of each value, which is denoted as the comprehensive probability of each value.

[0032] Preferably, adopt the uniform segmentation method to divide the data in each sliding window of each length into two segments.

[0033] In a second aspect, the present invention provides a data management system for IoT devices based on big data, adopting the following technical solutions:

[0034] A data management system for IoT devices based on big data includes: a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned data management method for IoT devices based on big data is implemented.

[0035] By adopting the above technical solutions, generate a computer program for the above-mentioned data management method for IoT devices based on big data, and store it in the memory to be loaded and executed by the processor, so as to manufacture a terminal device according to the memory and the processor, which is convenient to use.

[0036] The present invention has the following technical effects:

[0037] The present invention takes into account that the coding length is shorter when the interval shrinks rapidly in the early stage of coding and slowly in the later stage of coding. And the main factor affecting the interval shrinking speed is the segmentation ratio. Therefore, the coding length is reduced by adjusting the segmentation ratio.

[0038] Further, when adjusting the segmentation ratio, considering that if the data of a certain type of value is mainly distributed at the front side of the sequence, the segmentation ratio of the data of this type of value can be reduced to increase the shrinking speed of the early stage interval; if the data of a certain type of value is mainly distributed at the rear side of the sequence, the segmentation ratio of the data of this type of value can be increased to reduce the shrinking speed of the later stage interval, so as to achieve accurate adjustment of the segmentation ratio.

[0039] Further, considering that the segmentation ratio is related to the position of the data of various types of values in the sequence, the position of the data of various types of values in the sequence is accurately described by analyzing the degree of distribution preposition of the data of various types of values.

[0040] Further, when calculating the degree of distribution preposition, the position of the data of various types of values in the sequence is accurately reflected by introducing the center of gravity of various types of values and the distance between the centers of gravity. Description of the Drawings

[0041] By referring to the following detailed description with reference to the drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become easily understandable. In the drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0042] Figure 1 is a flowchart of the method in a data management method for Internet of Things devices based on big data according to an embodiment of the present invention. Detailed Embodiments

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0044] It should be understood that when the claims, specifications, and drawings of the present invention use terms such as "first" and "second", they are only used to distinguish different objects and are not used to describe a specific order. The terms "comprising" and "including" used in the specifications and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0045] An embodiment of the present invention discloses a data management method for Internet of Things devices based on big data. Refer to Figure 1 , including steps S1 - S5:

[0046] S1: Obtain the data sequence of the Internet of Things devices.

[0047] Specifically, obtain the data of the Internet of Things devices at each moment, and arrange the data of the Internet of Things devices at all moments in time sequence to obtain the data sequence of the Internet of Things devices. In this embodiment, the control data of the smart home is used as the data of the Internet of Things devices. In other embodiments, other data can be used as the data of the Internet of Things devices, and this embodiment does not make specific limitations.

[0048] S2: Obtain the occurrence probability of each value in the data sequence of the Internet of Things devices, denoted as the comprehensive probability, and obtain the distribution precedence degree of any value.

[0049] It should be noted that the encoding value obtained by encoding the sequence AAAAAAAAABBBBCCD using the arithmetic coding algorithm is: 0.0042322, and the encoding result obtained by encoding the sequence DCCBBBBAAAAAAAAA is: 0.99536; it can be seen by observation that the data types included in these two sequences are the same, both A, B, C, and D. The occurrence frequency of each data is also the same, and the only difference is the arrangement method. However, the encoding lengths differ by 2 due to the different arrangement methods. This is only a sequence of length 16, and its encoding length can differ by 2. For longer sequences, the encoding length difference is even greater. Therefore, in order to improve the compression effect, it is necessary to reduce the problem of the increase in the encoding length caused by the arrangement method. Through analysis, it can be known that the biggest difference between these two sequences is the arrangement difference of data with different frequencies. In the first sequence, the high-frequency data is arranged in the front and the low-frequency data is arranged in the back; in the second sequence, the low-frequency data is arranged in the front and the high-frequency data is arranged in the back. When the high-frequency data is arranged in the front, the reduction speed of the previous interval is slower, and the reduction speed of the later interval is faster, which will shrink the final interval to a smaller size; when the low-frequency data is arranged in the front, the reduction speed of the previous interval is faster, and the reduction speed of the later interval is slower, and it will not shrink the final interval to a smaller size. In order to reduce the problem of encoding growth caused by the arrangement method, it is necessary to increase the reduction speed of the previous interval and reduce the reduction speed of the later interval.

[0050] S20: Obtain the occurrence probability of each value in the data sequence of the Internet of Things devices, denoted as the comprehensive probability.

[0051] Preferably, as an example, obtaining the occurrence probability of each value in the data sequence of the Internet of Things devices, denoted as the comprehensive probability, includes:

[0052] Obtain the number of occurrences of each value in the data sequence of the Internet of Things devices, and divide the number of occurrences of each value by the number of the data sequence of the Internet of Things devices to obtain the occurrence probability of each value, denoted as the comprehensive probability of each value.

[0053] S21: Obtain the distribution front-degree of any value.

[0054] It should be noted that the segmentation ratio determines the reduction speed of the interval. To increase the reduction speed of the interval in the early stage and decrease the reduction speed of the interval in the later stage, it is necessary to lower the segmentation ratio of the data in the left segment of the main distribution and raise the segmentation ratio of the data in the right segment of the main distribution.

[0055] It should be further noted that to adjust the segmentation ratio, it is necessary to analyze the distribution front-degree of the data of each value. The distribution front-degree of the data of each value reflects the distribution of the data of each value in each segment.

[0056] Preferably, as an example, obtaining the distribution front-degree of any value includes:

[0057]

[0058] Among them, use sliding windows of several different lengths to slide on the Internet of Things device data sequence in turn, divide the data in each sliding window of each length into two segments, obtain the occurrence probability of this value in each segment and the distribution center of gravity of this value in each segment. represents the distance between the centers of gravity of this value in the two segments obtained from the j-th sliding window of the i-th length. 、 respectively represent the occurrence probability of this value in one of the segments obtained from the j-th sliding window of the i-th length and the occurrence probability of this value in the other segment. represents the number of sliding windows of the i-th length. represents the number of types of sliding window lengths.

[0059] It can be understood that reflects the probability difference between the two segments. The larger this value is, the more the data of this value is distributed in the left segment. Therefore, the greater the distribution front-degree of the data of this value. reflects the position of the data of this value in the segment. The larger this value is, the greater the degree of the data of this value at the front end or the back end of the sequence. Therefore when it is larger, the data is not only distributed in the left segment, but also more in the left part of the left segment. Therefore, the greater the distribution front-degree of the data of this value.

[0060] The above embodiments set the sliding window, segmentation, occurrence probability of this value in the segment and distribution center of gravity of this value in the segment for each length. Next, the obtaining methods of the sliding window, segmentation, occurrence probability of this value in the segment and distribution center of gravity of this value in the segment for each length will be described.

[0061] First, the method for obtaining sliding windows of each length includes:

[0062] Preset several lengths of sliding windows;

[0063] First, align the left end of the Internet of Things device data sequence with the left end of a sliding window of any length, and then slide the sliding window of this length on the Internet of Things device data sequence with a sliding step of 1 until the right end of the sliding window is aligned with the right end of the Internet of Things device data sequence, obtaining several sliding windows of this length. In this embodiment, the length of the sliding window is taken as each integer between, and in other embodiments, the length of the sliding window can be taken as other values, which are not specifically limited in this embodiment.

[0064] Then, the method for obtaining segments includes:

[0065] Evenly divide each sliding window of each length into two segments to obtain two segments.

[0066] Subsequently, the method for obtaining the occurrence probability of this value in the segment includes:

[0067] Obtain the number of occurrences of the data with this value in the segment, and divide the number of occurrences of the data with this value in the segment by the number of data in the segment to obtain the occurrence probability of this value in the segment.

[0068] Finally, the method for obtaining the distribution center of gravity of this value in the segment includes:

[0069] Obtain the positions of the data with this value in any segment, and obtain the geometric center of all the data with this value in this segment according to the positions of all the data with this value in this segment, which is recorded as the distribution center of gravity of this value in this segment.

[0070] S3: Calculate the adjustment coefficient for each value, where the adjustment coefficient is positively correlated with the degree of distribution preposition; set the segmentation ratio for each value according to the adjustment coefficient and the comprehensive probability, and the segmentation ratio is negatively correlated with the adjustment coefficient and negatively correlated with the comprehensive probability.

[0071] S30: Calculate the adjustment coefficient for each value.

[0072] Preferably, as an example, calculating the adjustment coefficient for each value includes:

[0073] Normalize the degree of distribution preposition of each value to obtain the adjustment coefficient for each value.

[0074] In this embodiment, the maximum-minimum normalization method is used for normalization, and other embodiments can use other normalization methods, which are not specifically limited in this embodiment.

[0075] S31: Set the segmentation ratio for each value according to the adjustment coefficient and the comprehensive probability.

[0076] It should be noted that the traditional arithmetic coding algorithm does not consider the data distribution situation and directly uses the data probability as the segmentation ratio. In order to reduce the phenomenon of the increase in the coding length caused by the data arrangement, the adjustment coefficient is used to adjust the segmentation ratio.

[0077] It should be further noted that in order to make the reduction speed of the early interval larger and the reduction speed of the later interval smaller; it is necessary to adjust down the segmentation ratio of the values with a large degree of distribution preposition and adjust up the segmentation ratio of the values with a small degree of distribution preposition.

[0078] Preferably, as an example, setting the segmentation ratio for each value according to the adjustment coefficient and the comprehensive probability includes:

[0079] Normalize each value's comprehensive probability after dividing it by the adjustment coefficient to obtain the segmentation ratio for each value.

[0080] S4: Perform coding control according to the segmentation ratio to achieve data storage management.

[0081] Preferably, as an example, performing coding control according to the segmentation ratio to achieve data storage management includes:

[0082] Arrange the segmentation ratios of all values in descending order. Denote the interval from 0 to 1 as the first-layer interval. According to the segmentation ratios of all values and the arrangement order, divide the first-layer interval into K intervals, denoted as the second-layer intervals, where K represents the number of value types; Obtain the second-layer target interval corresponding to the value of the first data in the IoT device data sequence. According to the segmentation ratios of all values and the arrangement order, divide the second-layer target interval into K intervals to obtain the third-layer intervals; and so on, until the final-layer intervals are obtained according to the penultimate data in the IoT device data sequence to complete the interval division. Randomly select a data in the final-layer target interval corresponding to the last data in the IoT device data sequence as the coding sequence.

[0083] Store the coding sequence and the segmentation ratio for each value as storage objects.

[0084] S5: Decode the coding sequence.

[0085] It should be noted that decoding the coding sequence according to the segmentation ratio is a prior art and will not be elaborated here.

[0086] An embodiment of the present invention also discloses a data management system for Internet of Things devices based on big data, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, a data management method for Internet of Things devices based on big data according to the present invention is implemented.

[0087] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface. Their settings and functions are known in the art, so they will not be described in detail here.

[0088] In the present invention, the aforementioned memory can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component. For example, a computer-readable storage medium can be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory, a dynamic random access memory, a static random access memory, an enhanced dynamic random access memory, a high-bandwidth memory, a hybrid memory cube, etc., or any other medium that can be used to store the required information and can be accessed by an application program, a module, or both. Any such computer storage medium can be a part of the device or accessible or connectable to the device.

[0089] Although this specification has shown and described multiple embodiments of the present invention, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art will think of many changes, alterations, and alternative ways without departing from the spirit and idea of the present invention. It should be understood that various alternative solutions to the embodiments of the present invention described herein can be adopted in the process of practicing the present invention.

[0090] The above are all preferred embodiments of the present invention. The protection scope of the present invention is not limited thereby. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A data management method for IoT devices based on big data, characterized in that: Includes steps: Get IoT device data sequence; The probability of occurrence of each value in the IoT device data sequence is recorded as the comprehensive probability, and the distribution pre-proportion of any value is obtained. , use several sliding windows of different lengths to slide on the IoT device data sequence in turn, divide the data in each sliding window of each length into two segments, obtain the occurrence probability of the value in each segment and the distribution center of the value in each segment, It represents the distance between the centroids of the values ​​in the two segments obtained by the j-th sliding window of the i-th length. , They represent the probability of occurrence of the value in one segment and the probability of occurrence of the value in another segment obtained by the j-th sliding window of the i-th length, respectively. represents the number of sliding windows of the i-th length, Indicates the number of types of sliding window lengths; Calculating the adjustment coefficient of each value, including: normalizing the distribution advance degree of each value to obtain the adjustment coefficient of each value; setting the segmentation ratio of each value according to the adjustment coefficient and the comprehensive probability, including: dividing the comprehensive probability of each value by the adjustment coefficient and then normalizing it to obtain the segmentation ratio of each value; performing encoding control according to the segmentation ratio to realize data storage management.

2. The data management method of IoT devices based on big data according to claim 1, characterized in that: The method of using sliding windows of different lengths to slide on the IoT device data sequence in sequence includes: Preset sliding windows of several lengths; First, align the left end of the IoT device data sequence with the left end of a sliding window of any length, and then use the sliding window of any length to slide on the IoT device data sequence with 1 as the sliding step to obtain several sliding windows of the length.

3. The data management method of IoT devices based on big data according to claim 1, characterized in that: The method for obtaining the distribution center of gravity of the value in each segment includes: The position of each data of this type of value in any segment is obtained, and the geometric center of all data of this type of value is obtained according to the position of all data of this type of value in the segment and recorded as the distribution center of gravity of this type of value in the segment.

4. The data management method of IoT devices based on big data according to claim 1, characterized in that: The encoding control according to the segmentation ratio includes: Arrange the segmentation ratios of all kinds of values ​​in descending order, record the interval from 0 to 1 as the first-level interval, and divide the first-level interval into K intervals as the second-level interval according to the segmentation ratios and arrangement order of all kinds of values, where K represents the number of types of values; obtain the second-level interval of the value corresponding to the first data in the IoT device data sequence as the second-level target interval, and divide the second-level target interval into K intervals as the third-level interval according to the segmentation ratios and arrangement order of all kinds of values; and so on, until the interval division is completed according to the second-to-last data in the IoT device data sequence to obtain the final-level interval, and randomly select a data in the final-level target interval of the value corresponding to the last data in the IoT device data sequence as the encoding sequence.

5. The data management method of IoT devices based on big data according to claim 4 is characterized in that: The data storage management is implemented by: Stores the encoded sequence as a storage object.

6. The data management method of IoT devices based on big data according to claim 1, characterized in that: The occurrence probability of each value in the IoT device data sequence is recorded as the comprehensive probability, including: The number of occurrences of each value in the IoT device data sequence is obtained, and the number of occurrences of each value is divided by the number of IoT device data sequences to obtain the probability of occurrence of each value, which is recorded as the comprehensive probability of each value.

7. The data management method of IoT devices based on big data according to claim 1, characterized in that: The data in each sliding window of each length is divided into two segments in a uniform segmentation manner.

8. A data management system for IoT devices based on big data, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a data management method for an Internet of Things device based on big data is implemented according to any one of claims 1 to 7.

Images