An Internet of Things heterogeneous data storage method and system

Abstract

The present invention discloses an Internet of Things (IoT) heterogeneous data storage method and system, mainly related to the technical field of IoT data storage. The method includes the following steps: the central gateway obtains the processor information of the user's IoT devices; establishes a data storage model according to the types of stored data of the IoT devices; respectively establishes a tree model with the central gateway as the central node in the established data storage model; the central gateway analyzes and judges the processor information of the accessed IoT devices and the data stream passing through the central gateway; the central gateway allocates storage paths and storage models to the data stream according to the analysis and judgment results. The beneficial effects of the present invention are as follows: it can solve the problems of classification and time-sharing storage of various types of data in the IoT, and improve the data call efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things data storage, and specifically to a method and system for storing heterogeneous data in the Internet of Things. Background Art

[0002] Internet of Things heterogeneous data storage refers to the process of managing data of different types, formats, and sources in the Internet of Things by adopting effective storage strategies and technologies. With the wide application of Internet of Things technologies, the storage and sharing of massive heterogeneous data have become a key issue.

[0003] In response to the above problems, corresponding solutions have been proposed in the prior art. For example, in the disclosed patent CN201310198922.3, a unified storage method is adopted at the data interface, effectively supporting the storage of data from different sources and of different types; the HBase technology is used to expand the column-oriented storage mechanism, enabling the storage capacity to be horizontally scalable; the memory mapping file mechanism is adopted to effectively solve the efficiency of large file storage and management; the data redundancy storage technology is adopted to greatly improve the reliability and access performance of the system. However, due to the particularity of Internet of Things devices, the types of data continue to increase, and the call popularity of the same type of data at different times also varies. How to classify and store heterogeneous data at different times and of different types in the Internet of Things has become an urgent problem to be solved in Internet of Things technologies.

[0004] Therefore, there is an urgent need for a method and system for storing heterogeneous data in the Internet of Things to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for storing heterogeneous data in the Internet of Things, which can solve the problems of classifying and storing various types of data at different times in the Internet of Things and improve the data call efficiency.

[0006] To achieve the above object, the present invention is realized through the following technical solutions:

[0007] On the one hand, a method for storing heterogeneous data in the Internet of Things is provided, including the following steps:

[0008] S1: The central gateway obtains the processor information of the user's Internet of Things device, and the processor information of the Internet of Things device specifically includes: the types of stored data of the Internet of Things device, the processor model version, the memory size, and the network transmission protocol information;

[0009] S2: Establish a data storage model according to the types of stored data of the Internet of Things device, and the data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model;

[0010] S3: Establish a tree model with the central gateway as the central node in the established data storage model. The tree model includes: a central node, several child nodes, and several leaf nodes;

[0011] S4: The central gateway analyzes and judges the processor information of the Internet of Things devices accessing it and the data flow passing through the central gateway;

[0012] S5: The central gateway allocates storage paths and storage models for the data flow according to the analysis and judgment results in step S4.

[0013] Preferably, in step S1, when the central gateway obtains the processor information of the user's Internet of Things devices, specifically:

[0014] The central gateway establishes a long connection with the Internet of Things devices, and the central gateway obtains and updates the stored data types and processor information of the Internet of Things devices in real time through the long connection.

[0015] Preferably, in step S2, when establishing the data storage model, specifically:

[0016] The central gateway establishes a three - level data storage model according to the frequency of the data stored in the Internet of Things devices being called within the same time period. The three - level data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model.

[0017] Preferably, in step S3, when establishing the tree model with the central gateway as the central node, specifically:

[0018] The central gateway divides the Internet of Things devices into different groups according to the different types of data collected by the Internet of Things devices accessing it and the network transmission protocols they belong to, and uses the processor model version and memory size characteristics of the Internet of Things devices as the performance criteria as the basis for data division of child nodes and leaf nodes in the tree - shaped structure model, forming a tree - shaped structure with the central gateway as the central node.

[0019] Preferably, in step S4, when the central gateway analyzes and judges the processor information of the Internet of Things devices accessing it and the data flow passing through the central gateway, specifically:

[0020] Within the same time period, the central gateway marks the data as hot data, warm data, and cold data according to the frequency of the data stored in the Internet of Things devices being called;

[0021] The central gateway judges the processor information of the Internet of Things devices accessing it, and uses the Internet of Things devices with high performance as child nodes, and sets a maximum accessible threshold for the child nodes.

[0022] Preferably, the marking the data as hot data, warm data, and cold data specifically includes the following steps:

[0023] S41: Set the data call period and the types of data to be called in the Internet of Things device. Denote the data call period as n0, and the types of data to be called as the set U0 = {A1, A2,... A n}, and mark all the data in the set U0 of the types of data to be called in the call period n0 as hot data;

[0024] S42: Denote the next call period after the call period n0 as n1, and the types of data to be called in the call period n1 as the set U1:

[0025] If U1 = U0, that is, U1 = {A1, A2,... A n}, then mark all the data in U1 as hot data;

[0026] If U1 ≠ U0, and the number of data types in U1 is less than or less than the number of data types in U0, then mark the data types in U1 that are the same as those in U0 as hot data, and mark the data types in U0 that are different from those in U1 as warm data;

[0027] S43: Denote the next call period after the call period n1 as n2, and the data set to be called in the call period n2 as U2:

[0028] If U2 = U0 = U1, that is, U2 = {A1, A2,... A n}, then mark all the data in U2 as hot data;

[0029] If U2 ≠ U0 ≠ U1, and the number of data types in U2 is less than U1 and less than U0 at the same time, then mark the data types in U2 that are the same as those in U1 and U0 as hot data, mark the data types in U1 that are different from those in U2 as warm data, and mark the data types in U0 that are different from both U1 and U2 as cold data;

[0030] If U2 ≠ U0 ≠ U1, and the number of data types in U2 is greater than U1 and greater than U0 at the same time, then mark the data types in U2 that are the same as those in U1 and U0 as hot data, and mark the data types in U2 that are different from both U1 and U2 as warm data;

[0031] S44: Follow steps S41 - S43, compare the data set in each call period after the call period n2 with the data sets in the previous call periods, and mark the data as hot data, warm data, and cold data in turn.

[0032] Preferably, the comparison of the data set in each call period after the call period n2 with the data sets in the previous call periods is specifically:

[0033] Based on the previous three call cycles, establish a hot data storage model, a warm data storage model, and a cold data storage model. At the start of the fourth call cycle, calculate the expected value of the same data type called within a week and set the maximum threshold and minimum threshold of the expected value. Data types greater than or equal to the maximum threshold are marked as hot data, data types less than or equal to the minimum threshold are marked as cold data, and the remaining data types are marked as warm data, and update the data types of hot data, warm data, and cold data.

[0034] Preferably, step S5 specifically includes the following steps:

[0035] S51: After the IoT device accesses the central gateway, preferentially mount to the sub-nodes in the hot data storage model. When the number of sub-nodes mounted in the hot data storage model reaches the maximum threshold, sequentially mount to the leaf nodes in the hot data storage model;

[0036] S52: Compare the call frequencies of the data collected from the newly accessed IoT device and the IoT data stored in the hot data storage model in step S51 at the same moment.

[0037] If the call frequency of the data collected from the newly accessed IoT device is greater than the IoT data stored in the hot data storage model in step S51, then transfer the IoT data stored in the hot data storage model in step S51 to the warm data storage model;

[0038] If the call frequency of the data collected from the newly accessed IoT device is equal to the IoT data stored in the hot data storage model in step S51, then compare the information of the newly accessed IoT device with the IoT device information in step S51, and the one with higher performance is used as the sub-node;

[0039] If the call frequency of the data collected from the newly accessed IoT device is less than the IoT data stored in the hot data storage model in step S51, then transfer the data collected from the newly accessed IoT device to the warm data storage model;

[0040] S53: Compare the call frequencies of the data collected from the IoT device accessed again and the IoT data stored in the warm data storage model in step S52 at the same moment.

[0041] If the call frequency of the data collected from the IoT device accessed again is greater than the IoT data stored in the warm data storage model in step S52, then transfer the IoT data stored in the warm data storage model in step S52 to the cold data storage model;

[0042] If the call frequency of the data collected by the Internet of Things device that is reconnected is equal to the Internet of Things data stored in the warm data storage model in step S52, then compare the information of the reconnected Internet of Things device with the information of the Internet of Things device in step S52, and select the one with higher performance as the sub-node;

[0043] If the call frequency of the data collected by the Internet of Things device that is reconnected is less than the Internet of Things data stored in the warm data storage model in step S52, then transfer the data collected by the reconnected Internet of Things device to the cold data storage model.

[0044] On the other hand, a heterogeneous Internet of Things data storage system is provided, including:

[0045] Data acquisition module: Specifically, it is the central gateway, which is used to acquire the types of stored data, processor model versions, memory sizes, and network transmission protocol information of Internet of Things devices;

[0046] Data processing module: Specifically, it is the central gateway, which is used to establish a three-level data storage model and a tree structure model based on the types of stored data and processor information of the acquired Internet of Things devices, and analyze and judge the data information and processor information of the Internet of Things devices;

[0047] Data storage module: Specifically, it is the Internet of Things device, which is used to store the data in the Internet of Things.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] The present invention provides a heterogeneous Internet of Things data storage method and system. First, based on the types and call frequencies of the data collected by the Internet of Things devices connected to the central gateway, the data is stored in a hierarchical manner, improving the data storage efficiency and thus the data call efficiency. Second, based on the models of the Internet of Things devices connected to the central gateway, the hierarchical Internet of Things devices are stored in a tree-like hierarchical manner, maximizing the solution to the problem of classifying and storing various types of data at different times. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of the heterogeneous Internet of Things data storage method of the present invention;

[0051] Figure 2 is a schematic structural diagram of the heterogeneous Internet of Things data storage system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0052] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by this application.

[0053] As Figure 1 shown, this embodiment provides an Internet of Things heterogeneous data storage method, including the following steps:

[0054] S1: The central gateway obtains the processor information of the user's Internet of Things device. The processor information of the Internet of Things device specifically includes: the types of stored data of the Internet of Things device, the processor model version, the memory size, and the network transmission protocol information;

[0055] S2: Establish a data storage model according to the types of stored data of the Internet of Things device. The data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model;

[0056] S3: Establish a tree model with the central gateway as the central node in the established data storage model. The tree model includes: a central node, several sub-nodes, and several leaf nodes;

[0057] S4: The central gateway analyzes and judges the processor information of the Internet of Things device accessing and the data stream passing through the central gateway;

[0058] S5: The central gateway allocates the storage path and storage model for the data stream according to the analysis and judgment results in step S4.

[0059] Among them, when the Internet of Things device is turned on, it actively sends a request to the central gateway and establishes a long connection with the central gateway to synchronously upload the processor model version, memory size, and network transmission protocol information of the Internet of Things device;

[0060] The central gateway establishes a three-level data storage model according to the network transmission protocol to which the Internet of Things device belongs and the data collection type. The three-level data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model. In this embodiment, since the data storage in the Internet of Things device has timeliness and will form data deposits over time, the three-level data storage model is pyramid-shaped, with the hot data storage model on the upper layer, the warm data storage model in the middle layer, and the cold data storage model on the bottom layer.

[0061] In the data storage model of each level, a tree model is established respectively, specifically:

[0062] The central gateway classifies the Internet of Things devices into different groups according to the types of data collected by the connected Internet of Things devices and the network transmission protocols they belong to. And based on the processor model version and memory size characteristics of the Internet of Things devices as performance criteria, it is used as the basis for data division of the child nodes and leaf nodes in the tree structure model, forming a tree structure with the central gateway as the central node. The central gateway analyzes and judges the processor information of the connected Internet of Things devices and the data flow passing through the central gateway. Specifically:

[0063] Within the same time period, the central gateway marks the data as hot data, warm data, and cold data according to the frequency of data being called in the Internet of Things devices. Specifically:

[0064] When the Internet of Things device is connected to the central gateway, the data stored in the Internet of Things device starts to be called. Set the data call cycle and the types of data being called in the Internet of Things device. In this embodiment, the first call cycle of the data being called in the Internet of Things device is denoted as n0, and n0 is 24 hours. Assume that in the call cycle n0, the dataset being called is denoted as U0, and U0 = {A1, A2, A3}. At this time, all data types in the dataset U0 are marked as hot data;

[0065] In the next call cycle n1 of the call cycle n0, that is, within the second 24 hours, the dataset being called in the Internet of Things device is denoted as U1. When U1 = U0 = {A1, A2, A3}, the data types being called in U1 are marked as hot data; when U1 = {A1, A2} < U0, mark A1 and A2 in U1 as hot data, and at the same time mark A3 in U0 as warm data; when U1 = {A1, A2, A3, A A 4}> U0, mark A1, A2, and A3 in U1 as hot data, and at the same time mark A4 in U1 as warm data;

[0066] In the next call cycle n2 of the call cycle n1, that is, within the third 24 hours, the dataset being called in the Internet of Things device is denoted as U2. When U2 = U0 = U1 = {A1, A2, A3}, the data types being called in U2 are marked as hot data; when (U2 = {A1, A2, A3, A4, A5})

[0067] When (U1 = {A1, A2, A3, A4}) > (u0 = {A1, A2, A3}), the same A1, A2, A3 in U2, U0, and U1 are marked as hot data, the same A4 in U2 and U1 is marked as warm data, and A5 in U2 is marked as cold data; when (U2 = {A1, A2, A3, A4}) = (U1 = {A1, A2, A3, A4}) > (U0 = {A1, A2, A3}), the same A1, A2, A3 in U2, U0, and U1 are marked as hot data, and the same A4 in U2 and U1 is marked as warm data; when (U2 = {A1, A2}) < (U0 = {A1, A2, A3}) < (U1 = {A1, A2, A3, A4}), A1 and A2 are marked as hot data, A3 is marked as warm data, and A4 is marked as cold data;

[0068] Based on the first three call cycles after an IoT device first accesses the central gateway, a hot data storage model, a warm data storage model, and a cold data storage model are established. As the call cycle continues, the data types called in the same IoT device will also change. Starting from the fourth call cycle of the IoT device, calculate the expected value of the same data type called within a week and set the maximum threshold and minimum threshold of the expected value. Data types greater than or equal to the maximum threshold are marked as hot data, data types less than or equal to the minimum threshold are marked as cold data, and the remaining data types are marked as warm data. And so on, calculate the expected value of the same data type called each subsequent week, and update the data types of hot data, warm data, and cold data;

[0069] The central gateway judges the processor information of the accessed IoT devices, and uses the IoT devices with high performance as sub - nodes, and sets the maximum access threshold for the sub - nodes.

[0070] Step S5 specifically includes:

[0071] S51: After the IoT device accesses the central gateway, it preferentially mounts to the sub - nodes in the hot data storage model. When the number of sub - nodes mounted in the hot data storage model reaches the maximum threshold, it is successively mounted to the leaf nodes in the hot data storage model;

[0072] S52: Compare the call frequency of the data collected from the newly accessed IoT device with the call frequency of the IoT data stored in the hot data storage model in step S51 at the same moment,

[0073] If the call frequency of the data collected from the newly accessed IoT device is greater than the IoT data stored in the hot data storage model in step S51, then transfer the IoT data stored in the hot data storage model in step S51 to the warm data storage model;

[0074] If the invocation frequency of the data collected by the newly connected Internet of Things device is equal to the Internet of Things data stored in the hot data storage model in step S51, then compare the information of the newly connected Internet of Things device with the information of the Internet of Things device in step S51, and select the one with higher performance as the child node;

[0075] If the invocation frequency of the data collected by the newly connected Internet of Things device is less than the Internet of Things data stored in the hot data storage model in step S51, then transfer the data collected by the newly connected Internet of Things device to the warm data storage model;

[0076] S53: Compare the invocation frequencies of the data collected by the Internet of Things device connected again and the Internet of Things data stored in the warm data storage model in step S52 at the same moment.

[0077] If the invocation frequency of the data collected by the Internet of Things device connected again is greater than the Internet of Things data stored in the warm data storage model in step S52, then transfer the Internet of Things data stored in the warm data storage model in step S52 to the cold data storage model;

[0078] If the invocation frequency of the data collected by the Internet of Things device connected again is equal to the Internet of Things data stored in the warm data storage model in step S52, then compare the information of the Internet of Things device connected again with the information of the Internet of Things device in step S52, and select the one with higher performance as the child node;

[0079] If the invocation frequency of the data collected by the Internet of Things device connected again is less than the Internet of Things data stored in the warm data storage model in step S52, then transfer the data collected by the Internet of Things device connected again to the cold data storage model.

[0080] This embodiment also provides a system based on the above Internet of Things heterogeneous data storage method, as Figure 2 shown, including:

[0081] Data acquisition module: Specifically, it is the central gateway, which is used to acquire the types of stored data, processor model versions, memory sizes, and network transmission protocol information of the Internet of Things devices;

[0082] Data processing module: Specifically, it is the central gateway, which is used to establish a three-level data storage model and a tree structure model according to the types of stored data and processor information of the Internet of Things devices, and analyze and judge the data information and processor information of the Internet of Things devices;

[0083] Data storage module: Specifically, it is the Internet of Things device, which is used to store the data in the Internet of Things.

[0084] The above is a specific description of the preferred embodiment of the present invention. However, the present invention is not limited to the described embodiment. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. An Internet of Things heterogeneous data storage method, characterized in that, Including the following steps: S1: The central gateway obtains the processor information of the user's Internet of Things (IoT) device. The processor information of the IoT device specifically includes: the types of stored data of the IoT device, the processor model version, the memory size, and the network transmission protocol information; S2: Establish a data storage model based on the types of stored data of the IoT device. The data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model; S3: Establish a tree model with the central gateway as the central node in the established data storage model. The tree model includes: a central node, several sub-nodes, and several leaf nodes; S4: The central gateway analyzes and judges the processor information of the connected IoT device and the data stream passing through the central gateway; S5: The central gateway allocates the storage path and storage model for the data stream according to the analysis and judgment results in step S4; Marking the data as hot data, warm data, and cold data specifically includes the following steps: S41: Set the data call period and the types of data to be called in the Internet of Things device. The data call period is denoted as , and the types of data to be called are denoted as a set . Mark all the data in the set of data types to be called during the call period as hot data; S42: Denote the next call period after the call period as . Denote the data types called during the call period as the set : If , that is , then mark all the data in as hot data; If , and the number of data types in is less than or equal to the number of data types in , then mark the data types in that are the same as those in as hot data, and mark the data types in that are different from those in as warm data; S43: Denote the next call period after the call period as . Denote the data set called during the call period as : If , that is , then all the data in will be marked as hot data; If , and the data types in are less than and at the same time less than then the data types in and that are the same as those in are marked as hot data, the data types in that are different from those in are marked as warm data, and the data types in and that are different from both are marked as cold data; If , and the data types in are greater than at the same time, then the data types in and are marked as hot data, and the data types in and are all marked as warm data; S44: Follow steps S41 - S43 to compare the data sets in each call cycle after the call cycle with the data sets in the previous call cycles, and mark the data as hot data, warm data, and cold data in sequence; For the call cycle Compare the data set in each call cycle after the call cycle with the data set in the previous call cycle, specifically: Based on the first three call cycles, establish a hot data storage model, a warm data storage model, and a cold data storage model. At the start of the fourth call cycle, calculate the expected value of the same data type called within a week and set the maximum threshold and minimum threshold of the expected value. The data types greater than or equal to the maximum threshold are marked as hot data, the data types less than or equal to the minimum threshold are marked as cold data, and the remaining data types are marked as warm data, and update the data types of hot data, warm data, and cold data; The step S5 specifically includes the following steps: S51: After the IoT device is connected to the central gateway, it preferentially mounts to the sub-nodes in the hot data storage model. When the number of mounted sub-nodes in the hot data storage model reaches the maximum threshold, it is successively mounted to the leaf nodes in the hot data storage model; S52: Compare the call frequency of the data collected from the newly connected IoT device with the IoT data stored in the hot data storage model in step S51 at the same moment, If the call frequency of the data collected from the newly connected IoT device is greater than the IoT data stored in the hot data storage model in step S51, then transfer the IoT data stored in the hot data storage model in step S51 to the warm data storage model; If the call frequency of the data collected from the newly connected IoT device is equal to the IoT data stored in the hot data storage model in step S51, then compare the information of the newly connected IoT device with the IoT device information in step S51, and the one with higher performance is used as the sub-node; If the call frequency of the data collected from the newly connected IoT device is less than the IoT data stored in the hot data storage model in step S51, then transfer the data collected from the newly connected IoT device to the warm data storage model; S53: Compare the call frequency of the data collected from the IoT device connected again with the IoT data stored in the warm data storage model in step S52 at the same moment, If the call frequency of the data collected by the IoT device reconnected is greater than the IoT data stored in the warm data storage model in step S52, then transfer the IoT data stored in the warm data storage model in step S52 to the cold data storage model; If the call frequency of the data collected by the IoT device reconnected is equal to the IoT data stored in the warm data storage model in step S52, then compare the information of the IoT device reconnected with the IoT device information in step S52, and select the one with higher performance as the child node; If the call frequency of the data collected by the IoT device reconnected is less than the IoT data stored in the warm data storage model in step S52, then transfer the data collected by the IoT device reconnected to the cold data storage model.

2. The method for storing heterogeneous data in the Internet of Things according to claim 1, wherein In step S1, the central gateway obtains the processor information of the user's IoT device, specifically: The central gateway establishes a long connection with the IoT device, and the central gateway obtains and updates the stored data types and processor information of the IoT device through the long connection in real time.

3. A method for storing heterogeneous data in the Internet of Things according to claim 1, characterized in that, In step S2, establish a data storage model, specifically: The central gateway establishes a three-level data storage model according to the call frequency of the data stored in the IoT device within the same time period. The three-level data storage model includes: a hot data storage model, a warm data storage model, and a cold data storage model.

4. A method for storing heterogeneous data in the Internet of Things according to claim 1, characterized in that, In step S3, establish a tree model with the central gateway as the central node, specifically: The central gateway divides the IoT devices into different groups according to the types of data collected by the IoT devices connected and the network transmission protocols they belong to, and uses the processor model version and memory size characteristics of the IoT devices as the performance criteria as the basis for data division of the child nodes and leaf nodes in the tree structure model, forming a tree structure with the central gateway as the central node.

5. A method for storing heterogeneous data in the Internet of Things according to claim 4, characterized in that In step S4, the central gateway analyzes and judges the processor information of the IoT device connected and the data stream passing through the central gateway, specifically: Within the same time period, the central gateway marks the data as hot data, warm data, and cold data according to the call frequency of the data stored in the IoT device; The central gateway judges the processor information of the IoT device connected and selects the IoT device with higher performance as the child node, and sets the maximum access threshold for the child node.

6. An Internet of Things heterogeneous data storage system, applicable to an Internet of Things heterogeneous data storage method as described in any one of claims 1-5, characterized in that, Including: Data acquisition module: specifically the central gateway, which is used to acquire the stored data types, processor model version, memory size, and network transmission protocol information of the IoT device; Data processing module: specifically the central gateway, which is used to establish a three-level data storage model and a tree structure model according to the acquired stored data types and processor information of the IoT device, and analyze and judge the data information and processor information of the IoT device; Data storage module: specifically the IoT device, which is used to store the data in the IoT.

Images