Internet of Things sensor data distributed storage method and middle division fusion system
By classifying and sharding the IoT sensor data, and fusion and migration in a multi-layer data chain according to the access frequency, the inefficiency problem of the distributed storage system of IoT sensor data is solved, and efficient storage and access.
Patent Information
- Application Number
- CN202411048147.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-26
AI Technical Summary
Existing distributed storage systems for IoT sensor data are difficult to meet the real-time and high access frequency requirements when expanding, resulting in inefficient access.
A distributed storage method for IoT sensor data is adopted, and data classification is obtained by obtaining sensor lists for classification, real-time data classification and identification and sharding processing is carried out, and distributed storage is carried out in a multi-layer data chain, and data is regularly or irregularly based on access frequency. Converge and migrate, optimize storage locations for increased access efficiency.
It realizes efficient distributed storage of sensor data, reasonably allocates storage resources, improves the access efficiency of high-frequency data, and solves the problem of distributed storage of sensor data.
Smart Images

Figure CN119025501A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed data storage, and more specifically, to a distributed storage method and a central fusion system for Internet of Things sensor data. Background Art
[0002] The Internet of Things refers to connecting any object to the network through information sensing devices according to agreed protocols. Objects exchange and communicate information through information transmission media to achieve intelligent identification, positioning, tracking, supervision and other functions. In the Internet of Things, sensors play a very important role and are the core link to achieve interconnection. Most of the existing conventional Internet of Things sensor data is stored locally. Due to the limitation of local memory resources, storage often cannot meet the demand, and it also brings great restrictions to the expansion of the Internet of Things.
[0003] Although the conventional distributed storage system storage method can solve the problem of data storage when the Internet of Things expands, the real-time and continuity of IoT sensor data, as well as the high access frequency of some sensor data, make it difficult for conventional distributed storage systems to be well applied and the access efficiency is low. A method for distributed storage of IoT sensor data is needed. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a distributed storage method for IoT sensor data and a central fusion system in view of the above-mentioned defects of the prior art.
[0005] The technical solution adopted by the present invention to solve the technical problem is:
[0006] A distributed storage method for IoT sensor data is constructed, which includes the following steps:
[0007] Step 1: Get a list of all sensors in the current IoT, classify the data types of the sensors according to the set classification strategy, and divide each data type into an independent temporary storage area;
[0008] Step 2: Acquire sensor data in the target IoT in real time, classify and identify the acquired sensor data, and store them in a temporary storage area corresponding to the data classification;
[0009] Step 3: Slice the sensor data in the temporary storage area, and the size of each data slice is the same;
[0010] Step 4: Distribute and store data shards in database units;
[0011] The database unit includes multiple layers of data ring chains that are arranged and distributed from the inside to the outside. Each data ring chain is composed of multiple data storage nodes linked in a ring shape. The storage efficiency of any data storage node on the inner data ring chain is better than that of any data storage node on the outer data ring chain. Any data storage node on the inner data ring chain establishes a connection with one or more data storage nodes on the outer data ring chain and backs up data on the outer data ring chain. The established connection also serves as a data migration channel.
[0012] Step 5: Perform statistical analysis on the access to the data shards at regular or irregular intervals, where the statistical items at least include whether there are data shards that are called multiple times at the same time and the frequency ranking of the data shards being called;
[0013] Step 6: Merge multiple data shards that are called simultaneously and exceed the set number of times in the analysis results into a new data shard for storage;
[0014] Step 7: Relying on the data migration channel, sort the data shards according to the frequency of their being called, and adjust the storage locations of multiple data shards regularly or irregularly, so that the data shards with higher calling frequencies are located on the innermost data ring chain.
[0015] The distributed storage method for IoT sensor data of the present invention further comprises the step of constructing a database unit:
[0016] Sensor data is classified according to the access frequency of sensor data in unit time, and the access frequency of sensor data in different classifications is in different segment ranges;
[0017] According to the requirements of each graded segment range on data storage efficiency, multiple data storage nodes that meet the requirements are selected;
[0018] Connect multiple data storage nodes corresponding to each level to form a data ring chain;
[0019] Reserving backup space in one or more data storage nodes in a next-level data ring chain in an adjacent data ring chain;
[0020] The data storage node with backup space is connected to one or more data storage nodes in the upper-level data ring chain.
[0021] In the distributed storage method for IoT sensor data of the present invention, the step of constructing a database unit further comprises:
[0022] A backup database is additionally configured for the lowest level data ring chain to facilitate data backup of one or more data storage nodes of the lowest level data ring chain.
[0023] The distributed storage method for IoT sensor data of the present invention, wherein the method for distributing and storing the data slices in the database unit is as follows: the data slices are stored in a data ring chain located in the middle of the existence space;
[0024] After the data shards are stored, the data shards are merged and migrated according to steps five to seven, and their positions in the database unit are adjusted.
[0025] The distributed storage method for IoT sensor data of the present invention, wherein when data fusion is performed, the backup data in the corresponding backup space remains unchanged;
[0026] When migrating data, data in the backup space is migrated at the same time.
[0027] The distributed storage method for IoT sensor data of the present invention comprises the following steps: the data types corresponding to the multiple data storage nodes of each data ring chain are not completely the same, and the data types corresponding to the multiple data storage nodes need to completely cover the data types of the data slices.
[0028] The distributed storage method for IoT sensor data of the present invention, wherein in the step one, the setting classification strategy includes classification by data generation or modification time period, classification by sensor type, classification by IoT device and classification by custom project.
[0029] A middle-division fusion system is applied to the above-mentioned distributed storage method of IoT sensor data, wherein the system includes a data preprocessing unit, a distributed storage unit and a database unit;
[0030] The data preprocessing unit is used to obtain a list of all sensors in the current Internet of Things, classify the data types of the sensors according to the set classification strategy, and divide each data type classification into an independent temporary storage area; obtain sensor data in the target Internet of Things in real time, classify and identify the obtained sensor data, and store it in a temporary storage area corresponding to the data classification;
[0031] The distributed storage unit is used to perform slicing processing on the sensor data in the temporary storage area, and each data slice has the same size; and distribute the data slices to the database unit;
[0032] The database unit comprises a plurality of data ring chains which are arranged and distributed from the inside to the outside, each data ring chain is composed of a plurality of data storage nodes which are linked in a ring shape, and the storage efficiency of any data storage node on the inner data ring chain is better than that of any data storage node on the outer data ring chain, and any data storage node on the inner data ring chain establishes a connection with one or more data storage nodes on the outer data ring chain and backs up data on the outer data ring chain, and the established connection also serves as a data migration channel;
[0033] The distributed storage unit also includes a data access statistics module, a data fusion module and a data migration module;
[0034] The data access statistics module is used to perform statistical analysis on the access to data shards at regular or irregular intervals, and the statistical items include whether there are data shards that are called multiple times at the same time and the frequency ranking of the data shards being called;
[0035] The data fusion module is used to fuse multiple data slices that are called simultaneously and exceed a set number of times in the analysis results into a new data slice for storage;
[0036] The data migration module is used to rely on the data migration channel, sort the data slices according to the frequency of being called, and adjust the storage positions of multiple data slices regularly or irregularly, so that the data slices with higher calling frequency are located on the innermost data ring chain.
[0037] The beneficial effects of the present invention are as follows: by applying the method of the present application, various sensor data are classified and sliced respectively, and the sizes are kept consistent after slicing to ensure uniformity during distributed storage, and at the same time, data ring chains with different weights are constructed, and fusion and migration operations are performed in the data ring chain according to the access situation of the sliced data, so as to achieve reasonable allocation of storage resources, ensure distributed storage while taking into account the access efficiency of high-frequency sensor data, thereby solving the problems encountered in the distributed storage of sensor data in the current Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:
[0039] Figure 1 This is a flow chart of a distributed storage method for IoT sensor data according to a preferred embodiment of the present invention;
[0040] Figure 2It is a principle block diagram of the center-separation fusion system of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.
[0042] The distributed storage method of IoT sensor data in a preferred embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:
[0043] S01: Obtain a list of all sensors in the current Internet of Things, classify the data types of the sensors according to the set classification strategy, and divide each data type into an independent temporary storage area;
[0044] S02: Acquire sensor data in the target IoT in real time, classify and identify the acquired sensor data, and store them in a temporary storage area corresponding to the data classification;
[0045] S03: Slice the sensor data in the temporary storage area, and each data slice has the same size;
[0046] S04: Distribute and store the data shards in the database unit;
[0047] The database unit includes multiple layers of data ring chains that are arranged and distributed from the inside to the outside. Each data ring chain is composed of multiple data storage nodes linked in a ring shape. The storage efficiency of any data storage node on the inner data ring chain is better than that of any data storage node on the outer data ring chain. Any data storage node on the inner data ring chain establishes a connection with one or more data storage nodes on the outer data ring chain and backs up data on the outer data ring chain. The established connection also serves as a data migration channel.
[0048] S05: performing statistical analysis on the access to the data shards at regular or irregular intervals, wherein the statistical items at least include whether there are data shards that are called multiple times at the same time and the frequency ranking of the calls to the data shards;
[0049] S06: merging multiple data shards that are called simultaneously and exceed the set number of times in the analysis results into a new data shard for storage;
[0050] S07: Relying on the data migration channel, the data shards are sorted according to the frequency of being called, and the storage locations of multiple data shards are adjusted regularly or irregularly, so that the data shards with higher calling frequencies are located on the innermost data ring chain;
[0051] By applying the method of the present application, various sensor data are classified and sharded respectively. After sharding, the sizes are kept consistent to ensure uniformity during distributed storage. At the same time, data ring chains with different weights are constructed, and fusion and migration operations are performed in the data ring chain according to the access situation of the sharded data, so as to achieve reasonable allocation of storage resources, ensure distributed storage while taking into account the access efficiency of high-frequency sensor data, thereby solving the problems encountered in the distributed storage of sensor data in the current Internet of Things.
[0052] The database unit can be constructed in the following ways:
[0053] Sensor data is classified according to the access frequency of sensor data in unit time, and the access frequency of sensor data in different classifications is in different segment ranges;
[0054] According to the requirements of each graded segment range on data storage efficiency, multiple data storage nodes that meet the requirements are selected;
[0055] Connect multiple data storage nodes corresponding to each level to form a data ring chain;
[0056] Reserving backup space in one or more data storage nodes in a next-level data ring chain in an adjacent data ring chain;
[0057] Connecting the data storage node with backup space to one or more data storage nodes in the upper-level data ring chain;
[0058] An additional backup database is configured for the lowest level data ring chain to facilitate data backup at one or more data storage nodes of the lowest level data ring chain;
[0059] The structure of the database unit provides the hardware foundation for data storage, fusion, redundant backup and migration. Of course, in distributed storage, the location of each data storage node in the above database unit is discrete. Through the above relationship and connection method, an overall database system is formed to meet the special access requirements of IoT sensor data.
[0060] For the parts of sensor data that need to be accessed frequently, it is necessary to configure more efficient data storage nodes through migration, and vice versa. Due to the real-time nature of sensor data, classification can well maintain the purity of sensor data of the same type. The fusion of data shards can solve the problem of separation of continuous data caused by previous shards.
[0061] Preferably, based on the above-mentioned special database unit structure, a method is adopted when distributing and storing the data shards in the database unit: the data shards are stored in a data ring chain located in the middle of the existence space; in this way, after the data shards are stored, the data shards are merged and migrated according to steps five to seven, and the positions in the database unit are adjusted.
[0062] At the same time, when data fusion is performed, the backup data in the corresponding backup space remains unchanged; when data migration is performed, the data in the backup space is migrated at the same time.
[0063] Preferably, for some sensor data that have requirements on the data type of the database, it can also be set that the data types corresponding to the multiple data storage nodes of each data ring chain are not exactly the same, and the data types corresponding to the multiple data storage nodes need to completely cover the data types of the data shards; in this way, the problem of data type incompatibility can be taken into account.
[0064] Preferably, in step one, setting the classification strategy includes classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and customizing project classification based on actual project needs.
[0065] A mid-point fusion system, such as Figure 2 As shown, the system includes a data preprocessing unit 10, a distributed storage unit 11 and a database unit 12;
[0066] The data preprocessing unit 10 is used to obtain a list of all sensors in the current Internet of Things, classify the data types of the sensors according to the set classification strategy, and divide each data type classification into an independent temporary storage area; obtain sensor data in the target Internet of Things in real time, classify and identify the obtained sensor data, and store it in the temporary storage area 100 corresponding to the data classification;
[0067] The distributed storage unit 11 is used to perform sharding processing on the sensor data in the temporary storage area 100, and each data shard has the same size; and the data shards are distributedly stored in the database unit 12;
[0068] The database unit 12 includes a multi-layer data ring chain 120 which is arranged and distributed from the inside to the outside. Each data ring chain is composed of a plurality of data storage nodes linked in a ring shape. The storage efficiency of any data storage node on the inner data ring chain is better than that of any data storage node on the outer data ring chain. Any data storage node on the inner data ring chain establishes a connection with one or more data storage nodes on the outer data ring chain and backs up data on the outer data ring chain. The established connection also serves as a data migration channel.
[0069] The distributed storage unit 11 further includes a data access statistics module 110, a data fusion module 111 and a data migration module 112;
[0070] The data access statistics module 110 is used to perform statistical analysis on the access of data shards at regular or irregular intervals. The statistical items include whether there are data shards that are called multiple times at the same time and the frequency ranking of the data shards being called;
[0071] The data fusion module 111 is used to fuse multiple data slices that are called simultaneously and exceed a set number of times in the analysis results into a new data slice for storage;
[0072] The data migration module 112 is used to rely on the data migration channel to sort the data slices according to the frequency of being called, and adjust the storage positions of multiple data slices regularly or irregularly, so that the data slices with higher calling frequency are located in the inner layer of the data ring chain;
[0073] By applying the method of the present application, various sensor data are classified and sharded respectively. After sharding, the sizes are kept consistent to ensure uniformity during distributed storage. At the same time, data ring chains with different weights are constructed, and fusion and migration operations are performed in the data ring chain according to the access situation of the sharded data, so as to achieve reasonable allocation of storage resources, ensure distributed storage while taking into account the access efficiency of high-frequency sensor data, thereby solving the problems encountered in the distributed storage of sensor data in the current Internet of Things.
[0074] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A distributed storage method for IoT sensor data, characterized in that: The following steps are involved: Step 1: Get a list of all sensors in the current IoT, classify the data types of the sensors according to the set classification strategy, and divide each data type into an independent temporary storage area; Step 2: Acquire sensor data in the target IoT in real time, classify and identify the acquired sensor data, and store them in a temporary storage area corresponding to the data classification; Step 3: Slice the sensor data in the temporary storage area, and the size of each data slice is the same; Step 4: Distribute and store data shards in database units; The database unit includes multiple layers of data ring chains that are arranged and distributed from the inside to the outside. Each data ring chain is composed of multiple data storage nodes linked in a ring shape. The storage efficiency of any data storage node on the inner data ring chain is better than that of any data storage node on the outer data ring chain. Any data storage node on the inner data ring chain establishes a connection with one or more data storage nodes on the outer data ring chain and backs up data on the outer data ring chain. The established connection also serves as a data migration channel. Step 5: Perform statistical analysis on the access to the data shards at regular or irregular intervals, where the statistical items at least include whether there are data shards that are called multiple times at the same time and the frequency ranking of the data shards being called; Step 6: Merge multiple data shards that are called simultaneously and exceed the set number of times in the analysis results into a new data shard for storage; Step 7: Relying on the data migration channel, sort the data shards according to the frequency of their being called, and adjust the storage locations of multiple data shards regularly or irregularly, so that the data shards with higher calling frequencies are located on the innermost data ring chain.
2. The distributed storage method for IoT sensor data according to claim 1, characterized in that: The method further comprises the step of constructing a database unit: Sensor data is classified according to the access frequency of sensor data in unit time, and the access frequency of sensor data in different classifications is in different segment ranges; According to the requirements of each graded segment range on data storage efficiency, multiple data storage nodes that meet the requirements are selected; Connect multiple data storage nodes corresponding to each level to form a data ring chain; Reserving backup space in one or more data storage nodes in a next-level data ring chain in an adjacent data ring chain; The data storage node with backup space is connected to one or more data storage nodes in the upper-level data ring chain.
3. The distributed storage method for IoT sensor data according to claim 2, characterized in that: The step of constructing a database unit also includes: A backup database is additionally configured for the lowest level data ring chain to facilitate data backup of one or more data storage nodes of the lowest level data ring chain.
4. The distributed storage method for IoT sensor data according to claim 2, characterized in that: The method for distributing and storing the data shards in the database unit is as follows: the data shards are stored in a data ring chain located in the middle of the existence space; After the data shards are stored, the data shards are merged and migrated according to steps five to seven, and their positions in the database unit are adjusted.
5. The distributed storage method for IoT sensor data according to claim 4 is characterized in that: When data fusion is performed, the backup data in the corresponding backup space remains unchanged; When migrating data, data in the backup space is migrated at the same time.
6. The distributed storage method for IoT sensor data according to claim 1, characterized in that: The data types corresponding to the multiple data storage nodes of each of the data ring chains are not completely the same, and the data types corresponding to the multiple data storage nodes need to completely cover the data types of the data slices.
7. The distributed storage method for IoT sensor data according to claim 1, characterized in that: In the step 1, the setting of the classification strategy includes classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items.
8. A central split fusion system, applied to the distributed storage method for IoT sensor data as claimed in any one of claims 1 to 7, characterized in that: The system includes a data preprocessing unit, a distributed storage unit and a database unit; The data preprocessing unit is used to obtain a list of all sensors in the current Internet of Things, classify the data types of the sensors according to the set classification strategy, and divide each data type classification into an independent temporary storage area; obtain sensor data in the target Internet of Things in real time, classify and identify the obtained sensor data, and store it in a temporary storage area corresponding to the data classification; The distributed storage unit is used to perform slicing processing on the sensor data in the temporary storage area, and each data slice has the same size; and distribute the data slices to the database unit; The database unit comprises a plurality of data ring chains which are arranged and distributed from the inside to the outside, each data ring chain is composed of a plurality of data storage nodes which are linked in a ring shape, and the storage efficiency of any data storage node on the inner data ring chain is better than that of any data storage node on the outer data ring chain, and any data storage node on the inner data ring chain establishes a connection with one or more data storage nodes on the outer data ring chain and backs up data on the outer data ring chain, and the established connection also serves as a data migration channel; The distributed storage unit also includes a data access statistics module, a data fusion module and a data migration module; The data access statistics module is used to perform statistical analysis on the access to data shards at regular or irregular intervals, and the statistical items include whether there are data shards that are called multiple times at the same time and the frequency ranking of the data shards being called; The data fusion module is used to fuse multiple data slices that are called simultaneously and exceed a set number of times in the analysis results into a new data slice for storage; The data migration module is used to rely on the data migration channel, sort the data slices according to the frequency of being called, and adjust the storage positions of multiple data slices regularly or irregularly, so that the data slices with higher calling frequency are located on the innermost data ring chain.
