Internet of Things heterogeneous data storage method and system

Abstract

The invention discloses an internet-of-things heterogeneous data storage method and system, and mainly relates to the technical field of internet-of-things data storage. Comprising the following steps: a central gateway obtains processor information of user Internet of Things equipment; establishing a data storage model according to the storage data type of the Internet of Things equipment; tree models with the central gateway as a central node are established in the established data storage model; the central gateway analyzes and judges the processor information of the accessed Internet of Things equipment and the data flow passing through the central gateway; and the central gateway performs storage path distribution and storage model distribution on the data stream according to an analysis and judgment result. The method has the beneficial effects that the problems of classification and time-sharing storage of various types of data in the Internet of Things can be solved, and the data calling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things data storage, and in particular to an Internet of Things heterogeneous data storage method and system. Background Art

[0002] Heterogeneous data storage in IoT refers to the process of managing data of different types, formats and sources in IoT using effective storage strategies and technologies. With the widespread application of IoT technology, the storage and sharing of massive heterogeneous data has become a key issue.

[0003] In view of the above problems, corresponding solutions have been proposed in the prior art. For example, the patent CN201310198922.3 proposes to adopt a unified storage method in the data interface to effectively support the storage of data from different sources and different types; to use HBase technology to expand the column-oriented storage mechanism so that the storage capacity can be horizontally expanded; to adopt a memory-mapped file mechanism to effectively solve the efficiency of large file storage and management; to adopt data redundancy storage technology to greatly improve the reliability and access performance of the system. However, due to the particularity of IoT devices, the types of data are constantly increasing and the call popularity of the same data type at different times will also be different. How to classify and time-share heterogeneous data of different types and at different times in the IoT has become an urgent problem to be solved in IoT technology.

[0004] Therefore, there is an urgent need for an IoT heterogeneous data storage method and system to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for storing heterogeneous data in the Internet of Things, which can solve the problem of classification and time-sharing storage of various types of data in the Internet of Things and improve data call efficiency.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In one aspect, a method for storing heterogeneous data in IoT is provided, comprising the following steps:

[0008] S1: The central gateway obtains the processor information of the user's IoT device, wherein the processor information of the IoT device specifically includes: the type of storage data of the IoT device, the processor model version, the memory size, and the network transmission protocol information;

[0009] S2: Establishing a data storage model according to the type of storage data of the IoT device, wherein the data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model;

[0010] S3: establishing a tree model with the central gateway as the central node in the established data storage model, wherein the tree model includes: a central node, a plurality of child nodes and a plurality of leaf nodes;

[0011] S4: The central gateway analyzes and determines the processor information of the connected IoT device and the data flow passing through the central gateway;

[0012] S5: The central gateway allocates storage paths and storage models to the data stream according to the analysis and judgment results in step S4.

[0013] Preferably, in step S1, the central gateway obtains the processor information of the user's IoT device, specifically:

[0014] The central gateway establishes a persistent connection with the IoT device, through which it acquires and updates in real time the storage data types and processor information of the IoT device.

[0015] Preferably, in step S2, a data storage model is established, specifically:

[0016] The central gateway establishes a three-level data storage model according to the frequency at which the data stored in the IoT device is called in the same time period. The three-level data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model.

[0017] Preferably, in step S3, the establishment of a tree model with the central gateway as the central node is specifically as follows:

[0018] The central gateway divides the IoT devices into different groups according to the different types of data collected by the connected IoT devices and the network transmission protocols they belong to. The processor model version and memory size characteristics of the IoT devices are used as performance standards as the basis for data division of child nodes and leaf nodes in the tree structure model, forming a tree structure with the central gateway as the central node.

[0019] Preferably, in step S4, the central gateway analyzes and determines the processor information of the connected IoT device and the data flow passing through the central gateway, specifically:

[0020] In the same time period, the central gateway marks the data as hot data, warm data, and cold data based on the frequency of data stored in the IoT devices being called;

[0021] The central gateway determines the processor information of the connected IoT devices, selects the IoT devices with high performance as child nodes, and sets the maximum accessible threshold for the child nodes.

[0022] Preferably, the step of marking the data as hot data, warm data, and cold data specifically includes the following steps:

[0023] S41: Set the data call cycle and the type of data to be called in the IoT device, the data call cycle is recorded as n0, and the type of data to be called is recorded as a set U0 = {A1, A2, ...A n}, mark all data in the data type set U0 called in the calling cycle n0 as hot data;

[0024] S42: The next calling cycle after calling cycle n0 is recorded as n1, and the types of data called in calling cycle n1 are recorded as set U1:

[0025] If U1=U0, that is, U1={A1,A2,…A n}, then all data in U1 are marked as hot data;

[0026] If U1≠U0, and the data types in U1 are less than or equal to those in U0, then the data types in U1 that are the same as those in U0 are marked as hot data, and the data types in U0 that are different from those in U1 are marked as warm data;

[0027] S43: The next calling cycle after calling cycle n1 is recorded as n2, and the data set called in calling cycle n2 is recorded as U2:

[0028] If U2=U0=U1, that is, U2={A1,A2,…A n}, then all data in U2 are marked as hot data;

[0029] If U2≠U0≠U1, and the data types in U2 are smaller than those in U1 and U0, then the data types in U2 that are the same as those in U1 and U0 are marked as hot data, the data types in U1 that are different from those in U2 are marked as warm data, and the data types in U0 that are different from those in both U1 and U2 are marked as cold data;

[0030] If U2≠U0≠U1, and the data types in U2 are greater than those in U1 and U0, then the data types in U2 that are the same as those in U1 and U0 are marked as hot data, and the data types in U2 that are different from those in U1 and U2 are marked as warm data;

[0031] S44: Following steps S41-S43, the data set in each calling cycle after the calling cycle n2 is compared with the data set in the previous calling cycle, and the data is marked as hot data, warm data, and cold data in turn.

[0032] Preferably, the comparison of the data set in each calling cycle after the calling cycle n2 with the data set in the previous calling cycle is specifically as follows:

[0033] Based on the first three calling cycles, a hot data storage model, a warm data storage model, and a cold data storage model are established. Starting from the fourth calling cycle, the expected value of the same data type being called within a week is calculated and the maximum and minimum thresholds of the expected value are set. Data types greater than or equal to the maximum threshold are marked as hot data, data types less than or equal to the minimum threshold are marked as cold data, and the remaining data types are marked as warm data, and the data types of hot data, warm data, and cold data are updated.

[0034] Preferably, the step S5 specifically comprises the following steps:

[0035] S51: After the IoT device is connected to the central gateway, the child nodes in the hot data storage model are mounted first. When the number of mounted child nodes in the hot data storage model reaches a maximum threshold, the child nodes are mounted to the leaf nodes in the hot data storage model in sequence.

[0036] S52: Compare the call frequency of the data collected from the newly connected IoT device with the IoT data stored in the hot data storage model in step S51 at the same time.

[0037] If the calling frequency of the data collected in the newly connected IoT device is greater than the IoT data stored in the hot data storage model in step S51, the IoT data stored in the hot data storage model in step S51 is transferred to the warm data storage model;

[0038] If the call frequency of the data collected in the newly connected IoT device is equal to the IoT data stored in the hot data storage model in step S51, the information of the newly connected IoT device is compared with the information of the IoT device in step S51, and the one with higher performance is used as the child node;

[0039] If the call frequency of the data collected in the newly connected IoT device is less than the IoT data stored in the hot data storage model in step S51, the data collected in the newly connected IoT device is transferred to the warm data storage model;

[0040] S53: Compare the call frequency of the data collected from the IoT device that is connected again with the IoT data stored in the warm data storage model in step S52 at the same time.

[0041] If the calling frequency of the data collected in the IoT device that is connected again is greater than the IoT data stored in the warm data storage model in step S52, the IoT data stored in the warm data storage model in step S52 is transferred to the cold data storage model;

[0042] If the calling frequency of the data collected in the reconnected IoT device is equal to the IoT data stored in the warm data storage model in step S52, the information of the reconnected IoT device is compared with the information of the IoT device in step S52, and the one with higher performance is used as the child node;

[0043] If the calling frequency of the data collected in the reconnected IoT device is less than the IoT data stored in the warm data storage model in step S52, the data collected in the reconnected IoT device is transferred to the cold data storage model.

[0044] On the other hand, a heterogeneous data storage system for IoT is provided, comprising:

[0045] Data collection module: specifically, the central gateway, which is used to collect the storage data type, processor model version, memory size and network transmission protocol information of IoT devices;

[0046] Data processing module: specifically, the central gateway, which is used to establish a three-level data storage model and a tree structure model according to the storage data types and processor information of the IoT devices, and analyze and judge the data information and processor information of the IoT devices;

[0047] Data storage module: specifically, IoT devices, used to store data in the IoT.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] The present invention provides a method and system for storing heterogeneous data of the Internet of Things. Firstly, based on the types and calling frequencies of data collected by the Internet of Things devices connected to the central gateway, the data are stored in a hierarchical manner, thereby improving the efficiency of data storage and thus improving the efficiency of data calling. Secondly, based on the models of the Internet of Things devices connected to the central gateway, the hierarchical Internet of Things devices are stored in a tree-like manner, thereby solving the problem of classification and time-sharing storage of multiple types of data to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the IoT heterogeneous data storage method of the present invention;

[0051] Figure 2 It is a schematic diagram of the structure of the heterogeneous data storage system of the Internet of Things of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0053] like Figure 1 As shown, this embodiment provides a method for storing heterogeneous data in IoT, including the following steps:

[0054] S1: The central gateway obtains the processor information of the user's IoT device, wherein the processor information of the IoT device specifically includes: the type of storage data of the IoT device, the processor model version, the memory size, and the network transmission protocol information;

[0055] S2: Establishing a data storage model according to the type of storage data of the IoT device, wherein the data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model;

[0056] S3: establishing a tree model with the central gateway as the central node in the established data storage model, wherein the tree model includes: a central node, a plurality of child nodes and a plurality of leaf nodes;

[0057] S4: The central gateway analyzes and determines the processor information of the connected IoT device and the data flow passing through the central gateway;

[0058] S5: The central gateway allocates storage paths and storage models to the data stream according to the analysis and judgment results in step S4.

[0059] When the IoT device is turned on, it actively initiates a request to the central gateway and establishes a long connection with the central gateway, synchronously uploading the processor model version, memory size, and network transmission protocol information of the IoT device;

[0060] The central gateway establishes a three-level data storage model according to the network transmission protocol and data collection type of the IoT device. The three-level data storage model includes: a hot data storage model, a warm data storage model and a cold data storage model. In this embodiment, since the data storage in the IoT device is time-sensitive and data deposition will be formed over time, the three-level data storage model is pyramid-shaped, with the upper layer being the hot data storage model, the middle layer being the warm data storage model, and the bottom layer being the cold data storage model.

[0061] In the data storage model at each level, a tree model is established, specifically:

[0062] The central gateway classifies IoT devices into different groups according to the types of data collected by the connected IoT devices and the network transmission protocols they belong to. Moreover, based on the processor model version and memory size characteristics of the IoT devices, the performance standard is used as the basis for data division of the sub-nodes and leaf nodes in the tree structure model, forming a tree structure with the central gateway as the central node. The central gateway analyzes and judges the processor information of the connected IoT devices and the data flow passing through the central gateway. Specifically:

[0063] Within the same time period, the central gateway marks the data as hot data, warm data, and cold data according to the frequency of data being called in the IoT devices. Specifically:

[0064] When an IoT device is connected to the central gateway, the data stored in the IoT device starts to be called. Set the data call cycle and the types of data being called in the IoT device. In this embodiment, the first call cycle of the data being called in the IoT device is denoted as n0, and n0 is 24 hours. Assume that in the call cycle n0, the data set being called is denoted as U0, and U0 = {A1, A2, A3}. At this time, all data types in the data set U0 are marked as hot data;

[0065] In the next call cycle n1 after the call cycle n0, that is, within the second 24 hours, the data set being called in the IoT device is denoted as U1. When U1 = U0 = {A1, A2, A3}, the data types being called in U1 are marked as hot data; when U1 = {A1, A2} < U0, A1 and A2 in U1 are marked as hot data, and at the same time, A3 in U0 is marked as warm data; when U1 = {A1, A2, A3, A A}> U0, A1, A2, and A3 in U1 are marked as hot data, and at the same time, A4 in U1 is marked as warm data;

[0066] In the next call cycle n2 after the call cycle n1, that is, within the third 24 hours, the data set being called in the IoT device is denoted as U2. When U2 = U0 = U1 = {A1, A2, A3}, the data types being called in U2 are marked as hot data; when (U2 = {A1, A2, A3, A4, A5})

[0067] >(U1={A1,A2,A3,A4})>(u0={A1,A2,A3}), mark the same A1, A2, A3 in U2, U0, and U1 as hot data, mark the same A4 in U2 and U1 as warm data, and mark A5 in U2 as cold data; when (U2={A1,A2,A3,A4})=(U1={A1,A2,A3,A4})>(U0={ A1, A2, A3}), mark A1, A2, A3 that are the same in U2, U0, and U1 as hot data, and mark A4 that are the same in U2 and U1 as warm data; when (U2={A1, A2})<(U0={A1, A2, A3})<(U1={A1, A2, A3, A4}), mark A1 and A2 as hot data, mark A3 as warm data, and mark A4 as cold data;

[0068] Based on the first three call cycles after an IoT device is first connected to the central gateway, a hot data storage model, a warm data storage model, and a cold data storage model are established. As the call cycle continues, the data type called in the same IoT device will also change accordingly. Starting from the fourth call cycle of the IoT device, the expected value of the same data type being called within a week is calculated and the maximum and minimum thresholds of the expected value are set. The data types greater than or equal to the maximum threshold are marked as hot data, the data types less than or equal to the minimum threshold are marked as cold data, and the remaining data types are marked as warm data. By analogy, the expected value of the same data type being called each subsequent week is calculated, and the data types of hot data, warm data, and cold data are updated;

[0069] The central gateway determines the processor information of the connected IoT devices, selects the IoT devices with high performance as child nodes, and sets the maximum accessible threshold for the child nodes.

[0070] The step S5 specifically includes:

[0071] S51: After the IoT device is connected to the central gateway, the child nodes in the hot data storage model are mounted first. When the number of mounted child nodes in the hot data storage model reaches a maximum threshold, the child nodes are mounted to the leaf nodes in the hot data storage model in sequence.

[0072] S52: Compare the call frequency of the data collected from the newly connected IoT device with the IoT data stored in the hot data storage model in step S51 at the same time.

[0073] If the calling frequency of the data collected in the newly connected IoT device is greater than the IoT data stored in the hot data storage model in step S51, the IoT data stored in the hot data storage model in step S51 is transferred to the warm data storage model;

[0074] If the call frequency of the data collected in the newly connected IoT device is equal to the IoT data stored in the hot data storage model in step S51, the information of the newly connected IoT device is compared with the information of the IoT device in step S51, and the one with higher performance is used as the child node;

[0075] If the call frequency of the data collected in the newly connected IoT device is less than the IoT data stored in the hot data storage model in step S51, the data collected in the newly connected IoT device is transferred to the warm data storage model;

[0076] S53: Compare the call frequency of the data collected from the IoT device that is connected again with the IoT data stored in the warm data storage model in step S52 at the same time.

[0077] If the calling frequency of the data collected in the IoT device that is connected again is greater than the IoT data stored in the warm data storage model in step S52, the IoT data stored in the warm data storage model in step S52 is transferred to the cold data storage model;

[0078] If the calling frequency of the data collected in the reconnected IoT device is equal to the IoT data stored in the warm data storage model in step S52, the information of the reconnected IoT device is compared with the information of the IoT device in step S52, and the one with higher performance is used as the child node;

[0079] If the calling frequency of the data collected in the reconnected IoT device is less than the IoT data stored in the warm data storage model in step S52, the data collected in the reconnected IoT device is transferred to the cold data storage model.

[0080] This embodiment also provides a system based on the above-mentioned IoT heterogeneous data storage method, such as Figure 2 As shown, including:

[0081] Data collection module: specifically, the central gateway, which is used to collect the storage data type, processor model version, memory size and network transmission protocol information of IoT devices;

[0082] Data processing module: specifically, the central gateway, which is used to establish a three-level data storage model and a tree structure model according to the storage data types and processor information of the IoT devices, and analyze and judge the data information and processor information of the IoT devices;

[0083] Data storage module: specifically, IoT devices, used to store data in the IoT.

[0084] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for storing heterogeneous data in IoT, characterized in that: The following steps are involved: S1: The central gateway obtains the processor information of the user's IoT device, wherein the processor information of the IoT device specifically includes: the type of storage data of the IoT device, the processor model version, the memory size, and the network transmission protocol information; S2: Establishing a data storage model according to the type of storage data of the IoT device, wherein the data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model; S3: establishing a tree model with the central gateway as the central node in the established data storage model, wherein the tree model includes: a central node, a plurality of child nodes and a plurality of leaf nodes; S4: The central gateway analyzes and determines the processor information of the connected IoT device and the data flow passing through the central gateway; S5: The central gateway allocates storage paths and storage models to the data stream according to the analysis and judgment results in step S4.

2. The method for storing heterogeneous data in IoT according to claim 1, characterized in that: In step S1, the central gateway obtains the processor information of the user's IoT device, specifically: The central gateway establishes a persistent connection with the IoT device, through which it acquires and updates in real time the storage data types and processor information of the IoT device.

3. The method for storing heterogeneous data in IoT according to claim 1, characterized in that: In step S2, a data storage model is established, specifically: The central gateway establishes a three-level data storage model according to the frequency at which the data stored in the IoT device is called in the same time period. The three-level data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model.

4. The method for storing heterogeneous data in IoT according to claim 1, characterized in that: In step S3, the tree model with the central gateway as the central node is established, specifically: The central gateway divides the IoT devices into different groups according to the different types of data collected by the connected IoT devices and the network transmission protocols they belong to. The processor model version and memory size characteristics of the IoT devices are used as performance standards as the basis for data division of child nodes and leaf nodes in the tree structure model, forming a tree structure with the central gateway as the central node.

5. The method for storing heterogeneous data in IoT according to claim 4, characterized in that: In step S4, the central gateway analyzes and determines the processor information of the connected IoT device and the data flow through the central gateway, specifically: In the same time period, the central gateway marks the data as hot data, warm data, and cold data based on the frequency of data stored in the IoT devices being called; The central gateway determines the processor information of the connected IoT devices, selects the IoT devices with high performance as child nodes, and sets the maximum accessible threshold for the child nodes.

6. The method for storing heterogeneous data in the Internet of Things according to claim 5, characterized in that: The step of marking the data as hot data, warm data, and cold data specifically includes the following steps: S41: Set the data call cycle and the type of data to be called in the IoT device, the data call cycle is recorded as n0, and the type of data to be called is recorded as a set U0 = {A1, A2, ...A n }, mark all data in the data type set U0 called in the calling cycle n0 as hot data; S42: The next calling cycle after calling cycle n0 is recorded as n1, and the types of data called in calling cycle n1 are recorded as set U1: If U1=U0, that is, U1={A1,A2,…A n }, then all data in U1 are marked as hot data; If U1≠U0, and the data types in U1 are less than or equal to those in U0, then the data types in U1 that are the same as those in U0 are marked as hot data, and the data types in U0 that are different from those in U1 are marked as warm data; S43: The next calling cycle after calling cycle n1 is recorded as n2, and the data set called in calling cycle n2 is recorded as U2: If U2=U0=U1, that is, U2={A1,A2,…A n }, then all data in U2 are marked as hot data; If U2≠U0≠U1, and the data types in U2 are smaller than those in U1 and U0, then the data types in U2 that are the same as those in U1 and U0 are marked as hot data, the data types in U1 that are different from those in U2 are marked as warm data, and the data types in U0 that are different from those in both U1 and U2 are marked as cold data; If U2≠U0≠U1, and the data types in U2 are greater than those in U1 and U0, then the data types in U2 that are the same as those in U1 and U0 are marked as hot data, and the data types in U2 that are different from those in U1 and U2 are marked as warm data; S44: Following steps S41-S43, the data set in each calling cycle after the calling cycle n2 is compared with the data set in the previous calling cycle, and the data is marked as hot data, warm data, and cold data in turn.

7. The method for storing heterogeneous data in the Internet of Things according to claim 6, characterized in that: The comparison of the data set in each calling cycle after the calling cycle n2 with the data set in the previous calling cycle is specifically as follows: Based on the first three calling cycles, a hot data storage model, a warm data storage model, and a cold data storage model are established. Starting from the fourth calling cycle, the expected value of the same data type being called within a week is calculated and the maximum and minimum thresholds of the expected value are set. Data types greater than or equal to the maximum threshold are marked as hot data, data types less than or equal to the minimum threshold are marked as cold data, and the remaining data types are marked as warm data, and the data types of hot data, warm data, and cold data are updated.

8. The method for storing heterogeneous data in the Internet of Things according to claim 7, characterized in that: The step S5 specifically comprises the following steps: S51: After the IoT device is connected to the central gateway, the child nodes in the hot data storage model are mounted first. When the number of mounted child nodes in the hot data storage model reaches a maximum threshold, the child nodes are mounted to the leaf nodes in the hot data storage model in sequence. S52: Compare the call frequency of the data collected from the newly connected IoT device with the IoT data stored in the hot data storage model in step S51 at the same time. If the calling frequency of the data collected in the newly connected IoT device is greater than the IoT data stored in the hot data storage model in step S51, the IoT data stored in the hot data storage model in step S51 is transferred to the warm data storage model; If the call frequency of the data collected in the newly connected IoT device is equal to the IoT data stored in the hot data storage model in step S51, the information of the newly connected IoT device is compared with the information of the IoT device in step S51, and the one with higher performance is used as the child node; If the call frequency of the data collected in the newly connected IoT device is less than the IoT data stored in the hot data storage model in step S51, the data collected in the newly connected IoT device is transferred to the warm data storage model; S53: Compare the call frequency of the data collected from the IoT device that is connected again with the IoT data stored in the warm data storage model in step S52 at the same time. If the calling frequency of the data collected in the IoT device that is connected again is greater than the IoT data stored in the warm data storage model in step S52, the IoT data stored in the warm data storage model in step S52 is transferred to the cold data storage model; If the calling frequency of the data collected in the reconnected IoT device is equal to the IoT data stored in the warm data storage model in step S52, the information of the reconnected IoT device is compared with the information of the IoT device in step S52, and the one with higher performance is used as the child node; If the calling frequency of the data collected in the reconnected IoT device is less than the IoT data stored in the warm data storage model in step S52, the data collected in the reconnected IoT device is transferred to the cold data storage model.

9. An IoT heterogeneous data storage system, characterized in that: include: Data collection module: specifically, the central gateway, which is used to collect the storage data type, processor model version, memory size and network transmission protocol information of IoT devices; Data processing module: specifically, the central gateway, which is used to establish a three-level data storage model and a tree structure model according to the storage data types and processor information of the IoT devices, and analyze and judge the data information and processor information of the IoT devices; Data storage module: specifically, IoT devices, used to store data in the IoT.

Images