Internet of Things data migration method and halving fusion system

Abstract

The invention relates to an Internet of Things data migration method and a mid-division fusion system, and the method comprises the following steps: carrying out the data classification of data in a buffer database, and carrying out the fragmentation processing according to a set fragmentation size; the distributed storage system comprises a processing server and multiple stages of storage databases, and each stage of storage database corresponds to data fragments in different weight ranges; the processing server performs distributed storage on the plurality of data fragments in the sub-database into a multi-level storage database, and the accessed data fragments are subjected to weighted processing; screening each level of storage database, and migrating the data fragments which do not accord with the weight range of the current storage database to the storage database which accords with the weight range; according to the technical scheme, the data of the Internet of Things are pulled, classified and fragmented in real time, large-flow real-time data processing of the Internet of Things can be effectively dealt with, data fragments which are frequently accessed in each system are migrated, and the access efficiency of high-frequency access data in the Internet of Things can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data storage, and more specifically, to an Internet of Things data migration method and a central distribution and fusion system. Background Art

[0002] Distributed storage is a storage method that stores data in multiple nodes. Compared with traditional centralized storage, distributed storage distributes data on multiple nodes, and each node can store and access data independently. Different from traditional distributed data storage, the distributed storage of data in the Internet of Things has its own special requirements: 1. Data is real-time, not fixed data; 2. For some data with high access frequency, the data location needs to be adjusted to improve access efficiency. To solve the above problems, a data migration method and a central fusion system for the Internet of Things are needed. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an Internet of Things data migration method and a middle-division fusion system in view of the above-mentioned defects of the prior art.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A method for data migration of the Internet of Things is constructed, wherein the method comprises the following steps:

[0006] Real-time acquisition of data to be stored in the Internet of Things is temporarily stored in the buffer database; data in the buffer database is classified and stored in the sub-databases respectively;

[0007] Perform sharding of the data in the current sub-database according to the set sharding size on a regular or irregular basis;

[0008] A corresponding distributed storage system is built for each sub-database. The distributed storage system includes a processing server and a multi-level storage database. Each level of storage database corresponds to data shards with different weight ranges.

[0009] The processing server distributes and stores multiple data shards in the sub-database into a multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards;

[0010] Regularly or irregularly, screen each level of storage database and migrate data shards that do not meet the weight range of the current storage database to storage databases that match the weight range;

[0011] After migration, modify the data address accordingly.

[0012] In the IoT data migration method described in the present invention, each level of storage database includes multiple databases, each database has an identification attribute, and the identification attribute includes a variable for storing the weight range of the data slice.

[0013] The IoT data migration method of the present invention further comprises the steps of:

[0014] The processing server counts the remaining storage capacity in each level of storage database. If the remaining storage capacity of the target storage database to be migrated is insufficient during data sharding migration, a set number of databases in the first-level storage database with the largest remaining storage capacity will be divided into the target storage database by modifying the weight range.

[0015] The method for migrating Internet of Things data of the present invention comprises the following steps: when distributed storage is performed in an initial state, the weight of the data shard is 0, and the storage does not consider the weight range at this time; after each access, the data shard is weighted according to a weighted algorithm.

[0016] In the IoT data migration method of the present invention, the storage database with a higher value of the smaller endpoint of the weight range corresponds to a database with higher comprehensive performance.

[0017] The IoT data migration method of the present invention further comprises the steps of:

[0018] Delete the data shards that have been distributed stored in the sub-database.

[0019] The IoT data migration method of the present invention comprises the following classification rules: classification by data generation or modification time period, classification by sensor type, classification by IoT device, and classification by custom project.

[0020] A mid-split fusion system is applied to the above-mentioned IoT data migration method, wherein the system includes a data cache module and a distributed storage module: the data cache module includes a cache database, a data classification unit and a data sharding unit; the distributed storage module includes a plurality of distributed storage systems corresponding to each sub-database;

[0021] The buffer database is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit is used to classify the data in the buffer database and store them in the sub-databases respectively; the data slicing unit is used to slice the data in the current sub-database regularly or irregularly according to the set slicing size;

[0022] The distributed storage system includes a processing server and a multi-level storage database, each level of the storage database corresponds to data shards with different weight ranges; the processing server distributes and stores multiple data shards in the sub-database into the multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards; regularly or irregularly screens each level of the storage database, and migrates data shards that do not meet the weight range of the current storage database to a storage database that matches the weight range; and after migration, the data address is modified accordingly.

[0023] The beneficial effects of the present invention are as follows: by applying the method of the present application, IoT data can be pulled in real time, classified and processed in slices, which can effectively cope with the large-volume real-time data processing of the IoT. At the same time, multiple distributed storage systems can be configured, and frequently accessed data slices can be migrated in each system, which can effectively improve the access efficiency of high-frequency access data in the IoT. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work:

[0025] Figure 1 This is a flow chart of the IoT data migration method according to a preferred embodiment of the present invention;

[0026] Figure 2 It is a principle block diagram of the center-separation fusion system of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0028] The Internet of Things data migration method of the preferred embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0029] S01: Real-time acquisition of data to be stored in the Internet of Things and temporary storage in a buffer database; data classification of the data in the buffer database and storage in sub-databases respectively;

[0030] S02: Perform sharding processing on the data in the current sub-database according to the set sharding size on a regular or irregular basis;

[0031] S03: construct a corresponding distributed storage system for each sub-database, wherein the distributed storage system includes a processing server and a multi-level storage database, and each level of storage database corresponds to data shards with different weight ranges;

[0032] S04: The processing server performs distributed storage of multiple data shards in the sub-database into a multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards;

[0033] S05: Regularly or irregularly, screen each level of storage database, and migrate data shards that do not meet the weight range of the current storage database to storage databases that meet the weight range; modify the data address accordingly after migration;

[0034] By applying the methods and approaches of this application, IoT data can be pulled in real time, classified and processed in segments, which can effectively cope with the large-volume real-time data processing of the IoT. At the same time, multiple distributed storage systems can be configured, and frequently accessed data segments can be migrated in each system, which can effectively improve the access efficiency of high-frequency access data in the IoT.

[0035] Preferably, each level of storage database includes multiple databases, each database has an identification attribute, and the identification attribute includes a variable for storing the weight range of the data shard; it is very convenient to quickly limit and adjust which set of storage databases the database is in; it should be noted that this setting is a dynamic setting, and there will be changes in the database every time a screening is performed, so as to more flexibly apply relatively limited storage resources.

[0036] More specifically: the processing server counts the remaining storage capacity in each level of storage database. If the remaining storage capacity of the target storage database to be migrated is insufficient when performing data shard migration, a set number of databases in the first-level storage database with the largest remaining storage capacity will be divided into the target storage database by modifying the weight range; when distributed storage is performed in the initial state, the data shard weight is 0, and the storage does not consider the weight range at this time; after each access, the data shard is weighted according to a weighted algorithm; a storage database with a higher value at the smaller endpoint of the weight range corresponds to a database with higher comprehensive performance; resources are integrated in this way, which is highly intelligent and can automatically adjust data; the initial storage database hierarchy can be customized, and during the application process, the number of hierarchies can be further increased or decreased as needed, and the adjustment is also very convenient.

[0037] Preferably, the method further comprises the steps of:

[0038] The data shards that have been distributed and stored in the sub-database are deleted to cope with the infinite continuous data flow in the Internet of Things through limited data processing space.

[0039] Preferably, the classification rules for data classification include classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items; targeted adjustments can be made according to actual needs and there is no limitation on this.

[0040] A middle-point fusion system is applied to the above-mentioned IoT data migration method, such as Figure 2 As shown, the system includes a data cache module 1 and a distributed storage module 2: the data cache module 1 includes a cache database 10, a data classification unit 11 and a data sharding unit 12; the distributed storage module 2 includes a plurality of distributed storage systems 20 corresponding to each sub-database;

[0041] The buffer database 10 is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit 11 is used to classify the data in the buffer database and store them in the sub-databases respectively; the data slicing unit 12 is used to slice the data in the current sub-database regularly or irregularly according to the set slicing size;

[0042] The distributed storage system 20 includes a processing server 200 and a multi-level storage database 201, each level of storage database corresponds to data shards with different weight ranges; the processing server performs distributed storage of multiple data shards in the sub-database into the multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards; regularly or irregularly screens each level of storage database, and migrates data shards that do not meet the weight range of the current storage database to a storage database that meets the weight range; after migration, the data address is modified accordingly;

[0043] By using the system of the present application, IoT data can be pulled in real time, classified and processed in segments, which can effectively cope with the large-volume real-time data processing of the IoT. At the same time, multiple distributed storage systems can be configured, and frequently accessed data segments can be migrated in each system, which can effectively improve the access efficiency of high-frequency access data in the IoT.

[0044] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for migrating data from the Internet of Things, characterized in that: The method comprises the following steps: Real-time acquisition of data to be stored in the Internet of Things is temporarily stored in the buffer database; data in the buffer database is classified and stored in the sub-databases respectively; Perform sharding of the data in the current sub-database according to the set sharding size on a regular or irregular basis; A corresponding distributed storage system is built for each sub-database. The distributed storage system includes a processing server and a multi-level storage database. Each level of storage database corresponds to data shards with different weight ranges. The processing server distributes and stores multiple data shards in the sub-database into a multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards; Regularly or irregularly, screen each level of storage database and migrate data shards that do not meet the weight range of the current storage database to storage databases that match the weight range; After migration, modify the data address accordingly.

2. The method for migrating IoT data according to claim 1, characterized in that: Each level of storage database includes multiple databases, each database has an identification attribute, and the identification attribute includes a variable for storing the weight range of the data shard.

3. The method for migrating IoT data according to claim 2, characterized in that: The method further comprises the steps of: The processing server counts the remaining storage capacity in each level of storage database. If the remaining storage capacity of the target storage database to be migrated is insufficient during data sharding migration, a set number of databases in the first-level storage database with the largest remaining storage capacity will be divided into the target storage database by modifying the weight range.

4. The method for migrating IoT data according to claim 3, characterized in that: When distributed storage is performed in the initial state, the weight of the data shard is 0, and the storage does not consider the weight range at this time; after each access, the data shard is weighted according to the weighted algorithm.

5. The method for migrating IoT data according to claim 4, characterized in that: The storage database having a higher value at the smaller endpoint of the weight range corresponds to a database with higher overall performance.

6. The method for migrating IoT data according to claim 1, characterized in that: The method further comprises the steps of: Delete the data shards that have been distributed stored in the sub-database.

7. The method for migrating IoT data according to claim 1, characterized in that: The classification rules of the data classification include classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items.

8. A split-fusion system, applied to the IoT data migration method according to any one of claims 1 to 7, characterized in that: The system includes a data cache module and a distributed storage module: the data cache module includes a cache database, a data classification unit and a data sharding unit; the distributed storage module includes a plurality of distributed storage systems corresponding to each sub-database; The buffer database is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit is used to classify the data in the buffer database and store them in the sub-databases respectively; the data slicing unit is used to slice the data in the current sub-database regularly or irregularly according to the set slicing size; The distributed storage system includes a processing server and a multi-level storage database, each level of the storage database corresponds to data shards with different weight ranges; the processing server distributes and stores multiple data shards in the sub-database into the multi-level storage database, receives and processes external access requests, and performs weighted processing on the accessed data shards; regularly or irregularly screens each level of the storage database, and migrates data shards that do not meet the weight range of the current storage database to a storage database that matches the weight range; and after migration, the data address is modified accordingly.

Images