Internet of Things data recovery method and halving fusion system

Abstract

The invention relates to an Internet of Things data recovery method and a halving fusion system. The method comprises the following steps: acquiring to-be-stored data generated in the Internet of Things in real time, and temporarily storing the to-be-stored data in a cache database; performing data classification on the data, and respectively storing the data in sub-databases; performing fragmentation processing on the data in the current sub-database regularly or irregularly according to the set fragmentation size; a corresponding distributed storage system is constructed for each sub-database, each distributed storage system comprises a processing server, multiple levels of storage databases and multiple levels of redundant databases in one-to-one correspondence with the storage databases, and each level of storage database corresponds to data fragments in different weight ranges; the processing server performs distributed storage on the plurality of data fragments in the sub-database into a multi-level storage database, receives and processes an external access request, and performs weighting processing on the accessed data fragments; by applying the method provided by the invention, reliable guarantee is provided for distributed storage of the Internet of Things data, and meanwhile, storage resources are reasonably allocated.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data storage, and more specifically, to an Internet of Things data recovery method and a central distribution and fusion system. Background Art

[0002] Distributed storage is a storage method that stores data in multiple nodes. Compared with traditional centralized storage, distributed storage distributes data on multiple nodes, and each node can store and access data independently. Different from traditional distributed data storage, distributed storage of data in the Internet of Things has its own special requirements: 1. Data is real-time, not fixed data; 2. For some data with high access frequency, when data loss occurs, it is necessary to respond more quickly and restore in time to ensure the reliability of data application. To solve the above problems, a data recovery method and a central fusion system for the Internet of Things are needed. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an Internet of Things data recovery method and a middle-division fusion system in view of the above-mentioned defects of the prior art.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A method for recovering IoT data is constructed, wherein the method comprises the following steps:

[0006] Real-time acquisition of data to be stored in the IoT and temporary storage in the cache database; data classification in the cache database and storage in the sub-databases respectively;

[0007] Perform sharding of the data in the current sub-database according to the set sharding size on a regular or irregular basis;

[0008] A corresponding distributed storage system is constructed for each sub-database. The distributed storage system includes a processing server, a multi-level storage database, and a multi-level redundant database corresponding to the storage database. Each level of storage database corresponds to data shards with different weight ranges.

[0009] The redundant database is set with a backup strategy associated with the weight: the weight is divided into multiple levels, and the higher the weight, the higher the frequency of data verification and the greater the number of backups in the corresponding level;

[0010] The processing server distributes and stores multiple data shards in the sub-database into a multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards;

[0011] Screen each level of storage database regularly or irregularly, migrate data shards that do not meet the weight range of the current storage database to storage databases that match the weight range, and migrate the backup data in the redundant database accordingly;

[0012] After migration, the data address is modified accordingly, and the redundant database processes the backup data according to the backup strategy.

[0013] The IoT data recovery method of the present invention comprises: each level of storage database comprises a plurality of databases, each database has an identification attribute, and the identification attribute comprises a variable for storing a weight range of a data slice; each level of redundant database comprises a plurality of data redundant nodes corresponding to the databases one by one, and a processing module is provided in the data redundant node for receiving and sending processing of the data slices, and executing a storage backup strategy.

[0014] The IoT data recovery method of the present invention further comprises the steps of:

[0015] The processing server counts the remaining storage capacity in each level of storage database. If the remaining storage capacity of the target storage database to be migrated is insufficient during data sharding migration, a set number of databases in the first-level storage database with the largest remaining storage capacity will be divided into the target storage database by modifying the weight range.

[0016] The method for recovering Internet of Things data of the present invention, wherein, when distributed storage is performed in the initial state, the weight of the data shard is 0, and the weight range is not considered in storage at this time; after each access, the data shard is weighted according to a weighted algorithm; the smaller endpoint value of the weight range is higher, and the storage database corresponds to a database with higher comprehensive performance.

[0017] The IoT data recovery method of the present invention, wherein the redundant database processes the backup data according to the backup strategy using a method;

[0018] According to the migration path of the data shard, the processing module in the data node to be migrated sends the corresponding backup data, and the processing module in the data node of the target migration address receives the backup data;

[0019] Adjust the frequency of data shard verification and the number of backups according to the weight level of the data shard;

[0020] If there are multiple backups, they are stored in the standby node associated with the current data node.

[0021] The IoT data recovery method of the present invention further comprises the steps of:

[0022] Delete the data shards that have been distributed stored in the sub-database.

[0023] The IoT data recovery method of the present invention comprises the following classification rules: classification by data generation or modification time period, classification by sensor type, classification by IoT device, and classification by custom project.

[0024] A mid-split fusion system is applied to the above-mentioned IoT data recovery method, wherein the system includes: a data cache module and a distributed storage module: the data cache module includes a cache database, a data classification unit and a data sharding unit; the distributed storage module includes a plurality of distributed storage systems corresponding to each sub-database;

[0025] The cache database is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit is used to classify the data in the cache database and store them in the sub-databases respectively; the data slicing unit is used to slice the data in the current sub-database regularly or irregularly according to the set slicing size;

[0026] The distributed storage system includes a processing server, a multi-level storage database and a multi-level redundant database corresponding to the storage database, each level of the storage database corresponds to data slices with different weight ranges; a backup strategy associated with the weight is set in the redundant database: the weight is divided into multiple levels, and the higher the weight, the higher the frequency of data proofreading and the greater the number of backups in the corresponding level; the processing server distributes multiple data slices in the sub-database into the multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data slices; screens each level of the storage database regularly or irregularly, migrates the data slices that do not meet the weight range of the current storage database to the storage database that matches the weight range, and migrates the backup data in the redundant database accordingly; after the migration, the data address is modified accordingly, and the redundant database processes the backup data according to the backup strategy.

[0027] The beneficial effects of the present invention are as follows: by applying the method of the present application, IoT data is classified and pulled in real time, and the classified data is installed and processed in shards of set sizes, which can well cope with large-volume IoT data. At the same time, the data shards are weighted and migrated according to the frequency of access, which can automatically achieve the distinction of high-frequency access data and the optimization of storage resources. In addition, the high-frequency access data shards after distinction are further proofread and set according to the weights, and the number of backups is adjusted to ensure the reliability of high-frequency data, thereby providing reliable protection for the distributed storage of IoT data while reasonably allocating storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:

[0029] Figure 1 This is a flow chart of the IoT data recovery method according to a preferred embodiment of the present invention;

[0030] Figure 2 It is a principle block diagram of the center-separation fusion system of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0032] The Internet of Things data recovery method of the preferred embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0033] S01: Real-time acquisition of data to be stored in the IoT and temporary storage in the cache database; data classification in the cache database and storage in the sub-databases respectively; sharding the data in the current sub-database regularly or irregularly according to the set shard size;

[0034] S02: construct a corresponding distributed storage system for each sub-database, wherein the distributed storage system includes a processing server, a multi-level storage database, and a multi-level redundant database corresponding to the storage database, and each level of storage database corresponds to data shards with different weight ranges;

[0035] S03: A backup strategy associated with weights is set in the redundant database: the weights are divided into multiple levels, and the higher the weight, the higher the frequency of data verification and the greater the number of backups in the corresponding level;

[0036] S04: The processing server performs distributed storage of multiple data shards in the sub-database into a multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards;

[0037] S05: Screen each level of storage database regularly or irregularly, migrate data shards that do not meet the weight range of the current storage database to storage databases that meet the weight range, and migrate the backup data in the redundant database accordingly;

[0038] S06: After the migration, the data address is modified accordingly, and the redundant database processes the backup data according to the backup strategy;

[0039] By applying the method of the present application, IoT data is classified and pulled in real time, and the classified data is installed and processed in shards of set sizes, which can effectively cope with large-volume IoT data. At the same time, the data shards are weighted and migrated according to the frequency of access, which can automatically achieve the distinction of high-frequency access data and the optimization of storage resources. In addition, the high-frequency access data shards after distinction are further proofread and the backup quantity is adjusted according to the weights to ensure the reliability of high-frequency data, thereby providing reliable protection for the distributed storage of IoT data while reasonably allocating storage resources.

[0040] Preferably, each level of storage database includes multiple databases, each database has an identification attribute, and the identification attribute includes a variable for storing the weight range of the data shard; it is very convenient to quickly limit and adjust the storage database in which the database is located; it should be noted that this setting is a dynamic setting, and there will be changes in the database every time a screening is performed, so as to more flexibly apply relatively limited storage resources;

[0041] Correspondingly, each level of redundant database also includes multiple data redundant nodes corresponding to the databases one by one, and a processing module is set in the data redundant node for receiving and sending data fragments, and executing the storage backup strategy;

[0042] Using the redundant database composition structure described above, the redundant database processes the backup data according to the backup strategy using a method;

[0043] According to the migration path of the data shard, the processing module in the data node to be migrated sends the corresponding backup data, and the processing module in the data node of the target migration address receives the backup data;

[0044] Adjust the frequency of data shard verification and the number of backups according to the weight level of the data shard;

[0045] If there are multiple backups, they are stored in the standby node associated with the current data node;

[0046] The frequency of proofreading and the number of backups can be adjusted accordingly based on the specific data sharding weights, ensuring the accuracy of high-frequency access data in the Internet of Things under limited resources.

[0047] More specifically: the processing server counts the remaining storage capacity in each level of storage database. If the remaining storage capacity of the target storage database to be migrated is insufficient when performing data shard migration, a set number of databases in the first-level storage database with the largest remaining storage capacity will be divided into the target storage database by modifying the weight range; when distributed storage is performed in the initial state, the data shard weight is 0, and the storage does not consider the weight range at this time; after each access, the data shard is weighted according to a weighted algorithm; a storage database with a higher value at the smaller endpoint of the weight range corresponds to a database with higher comprehensive performance; resources are integrated in this way, which is highly intelligent and can automatically adjust data; the initial storage database hierarchy can be customized, and during the application process, the number of hierarchies can be further increased or decreased as needed, and the adjustment is also very convenient.

[0048] Preferably, the method further comprises the steps of:

[0049] The data shards that have been distributed and stored in the sub-database are deleted to cope with the infinite continuous data flow in the Internet of Things through limited data processing space.

[0050] Preferably, the classification rules for data classification include classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items; targeted adjustments can be made according to actual needs and there is no limitation on this.

[0051] A mid-point fusion system is applied to the above-mentioned IoT data recovery method, such as Figure 2 As shown, the system includes: a data cache module 1 and a distributed storage module 2: the data cache module 1 includes a cache database 10, a data classification unit 11 and a data sharding unit 12; the distributed storage module 2 includes a plurality of distributed storage systems 20 corresponding to each sub-database;

[0052] The cache database 10 is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit 11 is used to classify the data in the cache database and store them in the sub-databases respectively; the data slicing unit 12 is used to perform slicing processing on the data in the current sub-database according to the set slicing size regularly or irregularly;

[0053] The distributed storage system 20 includes a processing server 200, a multi-level storage database 201 and a multi-level redundant database 202 corresponding to the storage database. Each level of the storage database 201 corresponds to data slices with different weight ranges. The redundant database 202 is provided with a backup strategy associated with the weight: the weight is divided into multiple levels. The higher the weight, the higher the frequency of data proofreading and the greater the number of backups in the corresponding level. The processing server 200 performs distributed storage of multiple data slices in the sub-database into the multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data slices. The storage database of each level is screened regularly or irregularly, and the data slices that do not meet the weight range of the current storage database are migrated to the storage database that matches the weight range, and the backup data in the redundant database is migrated accordingly. After the migration, the data address is modified accordingly, and the redundant database processes the backup data according to the backup strategy.

[0054] By using the system of the present application, IoT data can be classified and pulled in real time, and the classified data can be installed and processed in shards of set sizes, which can effectively cope with large-volume IoT data. At the same time, data shards can be weighted and migrated according to the frequency of access, which can automatically achieve the distinction of high-frequency access data and the optimization of storage resources. In addition, the differentiated high-frequency access data shards are further proofread and the backup quantity is adjusted according to the weights to ensure the reliability of high-frequency data, thereby providing reliable protection for the distributed storage of IoT data while reasonably allocating storage resources.

[0055] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for recovering Internet of Things data, characterized in that: The method comprises the following steps: Real-time acquisition of data to be stored in the IoT and temporary storage in the cache database; data classification in the cache database and storage in the sub-databases respectively; Perform sharding of the data in the current sub-database according to the set sharding size on a regular or irregular basis; A corresponding distributed storage system is constructed for each sub-database. The distributed storage system includes a processing server, a multi-level storage database, and a multi-level redundant database corresponding to the storage database. Each level of storage database corresponds to data shards with different weight ranges. The redundant database is set with a backup strategy associated with the weight: the weight is divided into multiple levels, and the higher the weight, the higher the frequency of data verification and the greater the number of backups in the corresponding level; The processing server distributes and stores multiple data shards in the sub-database into a multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards; Screen each level of storage database regularly or irregularly, migrate data shards that do not meet the weight range of the current storage database to storage databases that match the weight range, and migrate the backup data in the redundant database accordingly; After migration, the data address is modified accordingly, and the redundant database processes the backup data according to the backup strategy.

2. The IoT data recovery method according to claim 1, characterized in that: Each level of storage database includes multiple databases, each database has an identification attribute, and the identification attribute includes a variable for storing the weight range of data shards; each level of redundant database includes multiple data redundant nodes corresponding to the databases one by one, and a processing module is set in the data redundant node for receiving and sending data shards, and executing storage backup strategies.

3. The IoT data recovery method according to claim 2, characterized in that: The method further comprises the steps of: The processing server counts the remaining storage capacity in each level of storage database. If the remaining storage capacity of the target storage database to be migrated is insufficient during data sharding migration, a set number of databases in the first-level storage database with the largest remaining storage capacity will be divided into the target storage database by modifying the weight range.

4. The IoT data recovery method according to claim 3, characterized in that: When distributed storage is performed in the initial state, the weight of the data shard is 0, and the storage does not consider the weight range at this time; after each access, the data shard is weighted according to the weighted algorithm; the smaller endpoint value of the weight range is higher, and the storage database corresponds to a database with higher comprehensive performance.

5. The IoT data recovery method according to claim 4, characterized in that: The redundant database processes the backup data according to the backup strategy; According to the migration path of the data shard, the processing module in the data node to be migrated sends the corresponding backup data, and the processing module in the data node of the target migration address receives the backup data; Adjust the frequency of data shard verification and the number of backups according to the weight level of the data shard; If there are multiple backups, they are stored in the standby node associated with the current data node.

6. The IoT data recovery method according to claim 1, characterized in that: The method further comprises the steps of: Delete the data shards that have been distributed stored in the sub-database.

7. The IoT data recovery method according to claim 1, characterized in that: The classification rules of the data classification include classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items.

8. A split-fusion system is applied to the IoT data recovery method according to any one of claims 1 to 7, characterized in that: The system comprises: a data cache module and a distributed storage module: the data cache module comprises a cache database, a data classification unit and a data sharding unit; the distributed storage module comprises a plurality of distributed storage systems corresponding to each sub-database; The cache database is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit is used to classify the data in the cache database and store them in the sub-databases respectively; the data slicing unit is used to slice the data in the current sub-database regularly or irregularly according to the set slicing size; The distributed storage system includes a processing server, a multi-level storage database and a multi-level redundant database corresponding to the storage database, each level of the storage database corresponds to data slices with different weight ranges; a backup strategy associated with the weight is set in the redundant database: the weight is divided into multiple levels, and the higher the weight, the higher the frequency of data proofreading and the greater the number of backups in the corresponding level; the processing server distributes multiple data slices in the sub-database into the multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data slices; screens each level of the storage database regularly or irregularly, migrates the data slices that do not meet the weight range of the current storage database to the storage database that matches the weight range, and migrates the backup data in the redundant database accordingly; after the migration, the data address is modified accordingly, and the redundant database processes the backup data according to the backup strategy.

Images