Internet of Things data storage method and halving fusion system

Abstract

The invention relates to an Internet of Things data storage method and a mid-division fusion system. The method comprises the following steps: setting storage partitions for storing data streams of different data types in a data classification database; the virtual distributed storage module carries out fragmentation processing on the data in each storage partition, and carries out virtual distributed storage on the data in a plurality of virtual storage units; sending the storage information of the plurality of virtual storage units to a distributed storage module outside the Internet of Things, and creating a distributed storage task; the distributed storage module performs data distributed storage simulation according to the distributed storage task and obtains a plurality of corresponding storage addresses; the virtual distributed storage module correspondingly associates the storage addresses with the virtual storage units, and the virtual distributed storage module directly stores the data fragments into the corresponding storage addresses; by applying the method provided by the invention, multiple times of data transfer is not needed, the storage efficiency is high, and a series of defects caused by multiple times of data transfer are also avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage of Internet of Things data, and more specifically, to an Internet of Things data storage method and a distributed fusion system. Background Art

[0002] Data distributed storage is a data storage technology that uses the disk space on each machine in the enterprise through the network and forms these distributed storage resources into a virtual storage device. Data is stored in various corners of the enterprise in a distributed manner.

[0003] In the Internet of Things, a large amount of network data interaction is generated every moment. At present, for the distributed storage of data in the Internet of Things, it is usually only possible to first pull and organize the data stream according to a certain strategy to form a relatively complete data packet, and then the generated data packet is distributed and stored in pieces through a distributed storage module. The data needs to be transferred and stored many times, and the storage is time-consuming and inefficient. It is also more prone to packet loss and data congestion. There is a need for an Internet of Things data storage method and a central fusion system that can solve the above defects. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an Internet of Things data storage method and a middle-division fusion system in view of the above-mentioned defects of the prior art.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] Construct a method for storing data in the Internet of Things, which includes the following steps:

[0007] A data classification database is constructed within the Internet of Things, and storage partitions are set in the data classification database to store data streams of different data types;

[0008] A virtual distributed storage module is built inside the Internet of Things. The virtual distributed storage module slices the data in each storage partition and performs virtual distributed storage to multiple virtual storage units.

[0009] Sending storage information of multiple virtual storage units to a distributed storage module outside the Internet of Things to create a distributed storage task;

[0010] The distributed storage module performs data distribution storage simulation according to the distributed storage task and obtains corresponding multiple storage addresses;

[0011] The virtual distributed storage module associates the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices in the corresponding storage address.

[0012] The IoT data storage method of the present invention, wherein the virtual distributed storage module performs sharding processing on the data in each storage partition using the following method:

[0013] The total amount of data in the storage partition is counted at set intervals;

[0014] The number of shards is determined based on the total amount of data and the set shard size. Data that is less than one shard is sharded into one shard.

[0015] After sharding is completed, the divided data shards are transferred to the shard storage area;

[0016] Delete the data in the current storage partition to continue receiving subsequent data.

[0017] The IoT data storage method of the present invention, wherein the virtual distributed storage to the virtual storage unit adopts the method:

[0018] Count the number of shards in the shard storage area;

[0019] Generate multiple virtual storage units according to the set distributed storage rules;

[0020] The shard information and the shard storage address are written into the virtual storage unit.

[0021] The Internet of Things data storage method of the present invention, wherein the sending of the storage information of the multiple virtual storage units to the distributed storage module outside the Internet of Things, and the creation of the distributed storage task adopts the method:

[0022] The virtual distributed storage module counts all virtual storage unit data at set intervals and sends the virtual storage unit data to the distributed storage module; the virtual storage unit data includes the number of data shards, the type of data shards and the size of data shards; the distributed storage module establishes a distributed storage task based on the number of data shards, the type of data shards and the size of data shards.

[0023] The IoT data storage method of the present invention, wherein the distributed storage module performs data distribution storage simulation according to the distributed storage task, and obtains the corresponding multiple storage addresses using:

[0024] The distributed storage module determines the number of storage databases according to the number of virtual storage units in the distributed storage task;

[0025] The distributed storage module determines the type of storage database according to the data shard type in the distributed storage task;

[0026] The distributed storage module selects a storage database that meets the storage space requirements according to the number of data shards and the size of data shards in the distributed storage task;

[0027] The distributed storage module screens storage databases that meet the conditions and obtains the addresses of storage databases that meet the requirements.

[0028] The IoT data storage method of the present invention, wherein the virtual distributed storage module associates the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices into the corresponding storage addresses using the following method:

[0029] The virtual distributed storage module establishes a corresponding association relationship between the storage address and the virtual storage unit according to the type of the storage database and the remaining storage space of the storage database;

[0030] The virtual distributed storage module stores the multiple data slices corresponding to the virtual storage unit in the storage database according to the above association relationship, and deletes the virtual storage unit.

[0031] The Internet of Things data storage method described in the present invention, wherein the data types include status data, feedback data, positioning data, personalized data, etc.

[0032] A middle-division fusion system is applied to the above-mentioned Internet of Things data storage method, wherein the system includes a data classification database and a virtual distributed storage module located inside the Internet of Things, and also includes a distributed storage module and multiple storage databases located outside the Internet of Things;

[0033] The data classification database is provided with storage partitions for storing data streams of different data types;

[0034] The virtual distributed storage module processes the data in each storage partition in slices and performs virtual distributed storage to multiple virtual storage units; and sends the storage information of multiple virtual storage units to the distributed storage module;

[0035] The distributed storage module creates a distributed storage task, performs data distribution storage simulation according to the distributed storage task, and obtains storage addresses of corresponding multiple storage databases;

[0036] The virtual distributed storage module is also used to associate storage addresses with virtual storage units, and the virtual distributed storage module directly stores data slices in corresponding storage addresses.

[0037] The beneficial effect of the present invention is that: by applying the method of the present application, IoT data is subdivided based on data type, and data sharding and virtual distributed storage simulation are performed, and the simulation results are used to generate storage tasks in a distributed system, and the tasks are executed to obtain storage addresses that meet the requirements, and then the virtual storage units are associated with the databases corresponding to the storage addresses. Then, the sharded data can be directly stored in the storage addresses to complete data distributed storage, without the need for multiple data transfers, with high storage efficiency, and avoiding a series of defects caused by multiple data transfers. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:

[0039] Figure 1 This is a flow chart of the Internet of Things data storage method according to a preferred embodiment of the present invention;

[0040] Figure 2 It is a principle block diagram of the center-separation fusion system of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0042] The Internet of Things data storage method of the preferred embodiment of the present invention is as follows Figure 1 As shown, the following steps are included:

[0043] S01: construct a data classification database within the Internet of Things, and set storage partitions in the data classification database for storing data streams of different data types;

[0044] S02: construct a virtual distributed storage module within the Internet of Things, which processes the data in each storage partition into shards and performs virtual distributed storage to multiple virtual storage units;

[0045] S03: Sending storage information of multiple virtual storage units to a distributed storage module outside the Internet of Things to create a distributed storage task;

[0046] S04: The distributed storage module performs data distribution storage simulation according to the distributed storage task, and obtains corresponding multiple storage addresses;

[0047] S05: The virtual distributed storage module associates the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices in the corresponding storage address;

[0048] By applying the method of the present application, IoT data is segmented based on data types, and data sharding and virtual distributed storage simulation are performed. The simulation results are used to generate storage tasks in the distributed system, and the tasks are executed to obtain storage addresses that meet the requirements. Then, the virtual storage units are associated with the databases corresponding to the storage addresses, and then the sharded data can be directly stored in the storage addresses to complete data distributed storage. Multiple data transfers are not required, the storage efficiency is high, and a series of defects caused by multiple data transfers are avoided.

[0049] Preferably, the virtual distributed storage module slices the data in each storage partition using the following method:

[0050] The total amount of data in the storage partition is counted at set intervals;

[0051] The number of shards is determined based on the total amount of data and the set shard size. Data that is less than one shard is sharded into one shard.

[0052] After sharding is completed, the divided data shards are transferred to the shard storage area;

[0053] Delete the data in the current storage partition to continue receiving subsequent data;

[0054] This approach can ensure the uniformity of data sharding and the continuous sharding of data to cope with the continuous large-scale data flow of the Internet of Things.

[0055] Preferably, the virtual distributed storage to the virtual storage unit adopts the method:

[0056] Count the number of shards in the shard storage area;

[0057] Generate multiple virtual storage units according to the set distributed storage rules;

[0058] Writing the shard information and the shard storage address into the virtual storage unit;

[0059] The virtual storage unit does not need to allocate too much storage space, it only needs to be able to record relevant information, and the resource occupancy rate is extremely low.

[0060] Preferably, the storage information of multiple virtual storage units is sent to a distributed storage module outside the Internet of Things, and a distributed storage task is created by:

[0061] The virtual distributed storage module counts all virtual storage unit data at set intervals and sends the virtual storage unit data to the distributed storage module; the virtual storage unit data includes the number of data slices, the type of data slices, and the size of data slices; the distributed storage module establishes a distributed storage task based on the number of data slices, the type of data slices, and the size of data slices;

[0062] Within a set time period, multiple tasks may be established, and they can be queued and processed in the order of task establishment time; the content of the tasks is directly related to the storage database information, and the computing power required for task processing will be extremely low; this is also a core feature that can cope with the continuous data stream processing of the Internet of Things.

[0063] Preferably, the distributed storage module performs data distribution storage simulation according to the distributed storage task, and obtains the corresponding multiple storage addresses using:

[0064] The distributed storage module determines the number of storage databases according to the number of virtual storage units in the distributed storage task;

[0065] The distributed storage module determines the type of storage database according to the data sharding type in the distributed storage task;

[0066] The distributed storage module selects storage databases that meet the storage space requirements based on the number and size of data shards in the distributed storage task;

[0067] Through the above three conditions, the distributed storage module screens the storage databases that meet the conditions and obtains the addresses of the storage databases that meet the requirements.

[0068] Preferably, the virtual distributed storage module associates the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices into the corresponding storage address using the method:

[0069] The virtual distributed storage module establishes a corresponding association relationship between the storage address and the virtual storage unit according to the type of the storage database and the remaining storage space of the storage database;

[0070] The virtual distributed storage module stores the multiple data slices corresponding to the virtual storage unit in the storage database according to the above association relationship, and deletes the virtual storage unit;

[0071] The association matching is easy and can quickly match the database that meets the requirements; it should be noted that the virtual storage unit here has no association with the database before the association matching is performed, which can reduce the complexity of the above task processing and improve the efficiency of task processing.

[0072] Preferably, the data types include status data, feedback data, positioning data, and personalized data, etc.; the data types may also be set according to actual needs, and there is no limitation on this.

[0073] A middle-point fusion system is applied to the above-mentioned IoT data storage method, such as Figure 2 As shown, the system includes a data classification database 100 and a virtual distributed storage module 101 located inside the Internet of Things, and also includes a distributed storage module 200 and multiple storage databases 201 located outside the Internet of Things;

[0074] The data classification database 100 is provided with storage partitions for storing data streams of different data types;

[0075] The virtual distributed storage module 101 performs sharding processing on the data in each storage partition and performs virtual distributed storage to multiple virtual storage units; and sends the storage information of multiple virtual storage units to the distributed storage module;

[0076] The distributed storage module 200 creates a distributed storage task, performs data distribution storage simulation according to the distributed storage task, and obtains storage addresses of corresponding multiple storage databases 201;

[0077] The virtual distributed storage module 101 is also used to associate the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices in the corresponding storage address;

[0078] The system of the present application segments IoT data based on data types, performs data sharding and virtual distributed storage simulation, uses the simulation results to generate storage tasks in the distributed system, executes the tasks to obtain storage addresses that meet the requirements, and then performs association matching between the virtual storage units and the databases corresponding to the storage addresses. The sharded data can then be directly stored in the storage addresses to complete data distributed storage, without the need for multiple data transfers, with high storage efficiency, and avoiding a series of defects caused by multiple data transfers.

[0079] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for storing data in the Internet of Things, characterized in that: The following steps are involved: A data classification database is constructed within the Internet of Things, and storage partitions are set in the data classification database to store data streams of different data types; A virtual distributed storage module is built inside the Internet of Things. The virtual distributed storage module slices the data in each storage partition and performs virtual distributed storage to multiple virtual storage units. Sending storage information of multiple virtual storage units to a distributed storage module outside the Internet of Things to create a distributed storage task; The distributed storage module performs data distribution storage simulation according to the distributed storage task and obtains corresponding multiple storage addresses; The virtual distributed storage module associates the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices in the corresponding storage address.

2. The IoT data storage method according to claim 1, characterized in that: The virtual distributed storage module performs sharding processing on the data in each storage partition using the following method: The total amount of data in the storage partition is counted at set intervals; The number of shards is determined based on the total amount of data and the set shard size. Data that is less than one shard is sharded into one shard. After sharding is completed, the divided data shards are transferred to the shard storage area; Delete the data in the current storage partition to continue receiving subsequent data.

3. The method for storing IoT data according to claim 2, characterized in that: The method for performing virtual distributed storage to the virtual storage unit is as follows: Count the number of shards in the shard storage area; Generate multiple virtual storage units according to the set distributed storage rules; The shard information and the shard storage address are written into the virtual storage unit.

4. The method for storing IoT data according to claim 3, characterized in that: The method for sending the storage information of multiple virtual storage units to the distributed storage module outside the Internet of Things and creating a distributed storage task adopts: The virtual distributed storage module counts all virtual storage unit data at set intervals and sends the virtual storage unit data to the distributed storage module; the virtual storage unit data includes the number of data shards, the type of data shards and the size of data shards; the distributed storage module establishes a distributed storage task based on the number of data shards, the type of data shards and the size of data shards.

5. The method for storing IoT data according to claim 4, characterized in that: The distributed storage module performs data distribution storage simulation according to the distributed storage task, and obtains corresponding multiple storage addresses using: The distributed storage module determines the number of storage databases according to the number of virtual storage units in the distributed storage task; The distributed storage module determines the type of storage database according to the data shard type in the distributed storage task; The distributed storage module selects a storage database that meets the storage space requirements according to the number of data shards and the size of data shards in the distributed storage task; The distributed storage module screens storage databases that meet the conditions and obtains the addresses of storage databases that meet the requirements.

6. The method for storing IoT data according to claim 5, characterized in that: The virtual distributed storage module associates the storage address with the virtual storage unit, and the virtual distributed storage module directly stores the data slices into the corresponding storage address using the method: The virtual distributed storage module establishes a corresponding association relationship between the storage address and the virtual storage unit according to the type of the storage database and the remaining storage space of the storage database; The virtual distributed storage module stores the multiple data slices corresponding to the virtual storage unit in the storage database according to the above association relationship, and deletes the virtual storage unit.

7. The IoT data storage method according to claim 1, characterized in that: The data types include status data, feedback data, positioning data and personalized data.

8. A split-fusion system, applied to the IoT data storage method according to any one of claims 1 to 7, characterized in that: The system includes a data classification database and a virtual distributed storage module located inside the Internet of Things, and also includes a distributed storage module and multiple storage databases located outside the Internet of Things; The data classification database is provided with storage partitions for storing data streams of different data types; The virtual distributed storage module performs sharding processing on the data in each storage partition and performs virtual distributed storage to multiple virtual storage units; Sending storage information of the plurality of virtual storage units to the distributed storage module; The distributed storage module creates a distributed storage task, performs data distribution storage simulation according to the distributed storage task, and obtains storage addresses of corresponding multiple storage databases; The virtual distributed storage module is also used to associate storage addresses with virtual storage units, and the virtual distributed storage module directly stores data slices in corresponding storage addresses.

Images