Internet of Things data fusion method and halving fusion system

Abstract

The invention relates to an Internet of Things data fusion method and a middle division fusion system, and the method comprises the following steps: carrying out the fragmentation processing of data in a sub-database, and endowing an identifier to a data fragment with a fusion analysis demand in the sub-database; a data distributed storage system is constructed, a fusion storage module is constructed in the data distributed storage system, and the fusion storage module comprises a plurality of fusion storage nodes and a plurality of sub-storage nodes; establishing a relation between the fusion storage node and the sub-storage nodes storing the corresponding data fragments; during access, the data distributed storage system receives an external fusion access request, and performs data fusion and feedback of fused data through the fusion storage node; according to the technical scheme, distributed storage of continuous large-flow data of the Internet of Things and a data fusion analysis request can be effectively coped with, and fusion analysis and data storage are carried out dispersedly, so that the data processing efficiency can be effectively improved, and meanwhile, the equipment load and the requirement for the equipment processing capacity are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data storage, and more specifically, to an Internet of Things data fusion method and a distributed fusion system. Background Art

[0002] Distributed storage is a storage method that stores data in multiple nodes. Compared with traditional centralized storage, distributed storage distributes data on multiple nodes, and each node can store and access data independently. Different from traditional data distributed storage, the distributed storage of data in the Internet of Things has its own special requirements: 1. Data is real-time, not fixed data; 2. There are certain data aggregation and analysis requirements between the data nodes of distributed storage. Traditional distributed storage technology can only perform segmented distributed storage of data in time periods. When data aggregation and analysis are required, data reading and fusion operations are required, which has low processing efficiency. Moreover, the computing fusion tasks are concentrated in a distributed system, which is very likely to cause data processing to be untimely and congestion. A data fusion method and a central fusion system for the Internet of Things that can better take into account and solve the above problems are needed. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an Internet of Things data fusion method and a central fusion system in view of the above-mentioned defects of the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is:

[0005] A method for fusion of IoT data is constructed, wherein the method comprises the following steps:

[0006] Real-time acquisition of data to be stored in the Internet of Things is temporarily stored in the buffer database; data in the buffer database is classified and stored in sub-databases with classification labels;

[0007] Perform sharding processing on the data in the current sub-database regularly or irregularly according to the set sharding size, and assign identifiers to the data shards in the sub-database that have fusion analysis requirements according to the set or input data fusion strategy;

[0008] Constructing a data distributed storage system corresponding to each sub-database, constructing a fusion storage module in the data distributed storage system, wherein the fusion storage module includes a plurality of fusion storage nodes and a plurality of sub-storage nodes;

[0009] Each fusion storage node stores one or more data fusion relationships; one or more data shards are stored in the child storage node; the fusion storage node establishes a connection with the child storage node storing the corresponding data shards;

[0010] During access, the data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion and feedback of fusion data through fusion storage nodes.

[0011] The IoT data fusion method of the present invention, wherein the method of assigning identifiers to data fragments in the sub-database with fusion analysis requirements according to the set or input data fusion strategy adopts:

[0012] Each data slice contains multiple data packets;

[0013] The data fusion strategy includes a list of data fusion requirements in units of data packets;

[0014] If a data packet listed in the data fusion requirement list exists in a data slice, the data slice is considered to be a data slice with fusion analysis requirements.

[0015] The Internet of Things data fusion method of the present invention, wherein each fusion storage node stores one or more data fusion relationships using the method:

[0016] The data fusion relationship corresponds to the fusion items in the data fusion requirement list;

[0017] Data packets that require data fusion in data shards are replaced with set characters;

[0018] Each data fusion relationship includes a plurality of setting characters.

[0019] In the IoT data fusion method of the present invention, the fusion storage node establishes a connection with the sub-storage node storing the corresponding data slices using the following method:

[0020] Multiple data shards in the same sub-database are distributed and stored in multiple sub-storage nodes;

[0021] With the set characters as matching items, multiple fusion storage nodes and multiple sub-storage nodes are connected.

[0022] The IoT data fusion method of the present invention, wherein the data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion through the fusion storage nodes and feedback of the fused data adopts the following method:

[0023] The data distributed storage system receives external fusion access requests;

[0024] Decomposing the fusion access request into multiple data packets, matching the data fusion relationship according to the set characters corresponding to the data packets, and then matching the fusion storage node having the data fusion relationship;

[0025] Sending a fusion instruction to the fusion storage node;

[0026] The fusion storage node pulls the data shards from the corresponding sub-storage node according to the fusion instruction, and performs fusion analysis and processing according to the set rules, and the processed data is fed back to the outside world.

[0027] The method for fusion of Internet of Things data of the present invention further comprises the steps of:

[0028] Delete the data shards that have been distributed stored in the sub-database.

[0029] The Internet of Things data fusion method described in the present invention, wherein the classification rules of the data classification include classification by data generation or modification time period, classification by sensor type, classification by equipment in the Internet of Things, and classification by custom items.

[0030] A middle-division fusion system is applied to the above-mentioned IoT data fusion method, wherein the system includes a data preprocessing module and a distributed storage module;

[0031] The data preprocessing module includes a buffer database, a data classification unit and a data sharding unit; the distributed storage module includes a plurality of data distributed storage systems;

[0032] The buffer database is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit is used to classify the data in the buffer database and store them in sub-databases with classification labels respectively; the data slicing unit is used to slice the data in the current sub-database regularly or irregularly according to the set slice size, and assign identifiers to the data slices in the sub-database that have fusion analysis requirements according to the set or input data fusion strategy;

[0033] A fusion storage module is constructed in the data distributed storage system, and the fusion storage module includes multiple fusion storage nodes and multiple sub-storage nodes; each fusion storage node stores one or more data fusion relationships; one or more data slices are stored in the sub-storage node; the fusion storage node establishes a connection with the sub-storage node storing the corresponding data slices; when accessing, the data distributed storage system receives external fusion access requests, matches the fusion storage nodes, and performs data fusion and feedback of the fused data through the fusion storage nodes.

[0034] The beneficial effect of the present invention is that: by applying the method of the present application, the large-flow Internet of Things is classified and processed in pieces to avoid front-end data congestion, and multiple data distributed storage systems are constructed based on the classification. Each data distributed storage system has an independent fusion storage module to match and process fusion requests. This method can effectively cope with the distributed storage of continuous large-flow data of the Internet of Things and the fusion analysis requests for data, and the fusion analysis and data storage are carried out in a decentralized manner, thereby effectively improving data processing efficiency while reducing equipment load and requirements for equipment processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:

[0036] Figure 1 This is a flow chart of the Internet of Things data fusion method according to a preferred embodiment of the present invention;

[0037] Figure 2 This is a principle block diagram of the Internet of Things data fusion system of a preferred embodiment of the present invention;

[0038] Figure 3 It is a topological diagram of multiple fusion storage nodes and multiple sub-storage nodes of the Internet of Things data fusion system of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0040] The Internet of Things data fusion method of the preferred embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0041] S01: Real-time acquisition of data to be stored in the Internet of Things and temporary storage in a buffer database; data classification in the buffer database and storage in sub-databases with classification labels;

[0042] S02: Perform sharding processing on the data in the current sub-database regularly or irregularly according to the set sharding size, and assigning identifiers to the data shards in the sub-database that have fusion analysis requirements according to the set or input data fusion strategy;

[0043] S03: constructing a data distributed storage system corresponding to each sub-database, and constructing a fusion storage module in the data distributed storage system, wherein the fusion storage module includes a plurality of fusion storage nodes and a plurality of sub-storage nodes;

[0044] S04: Each fusion storage node stores one or more data fusion relationships; one or more data slices are stored in the child storage node; the fusion storage node establishes a connection with the child storage node storing the corresponding data slice;

[0045] S05: During access, the data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion and feedback of fusion data through the fusion storage nodes;

[0046] By applying the method of the present application, large-volume IoT traffic is classified and processed in segments to avoid front-end data congestion, and multiple data distributed storage systems are constructed based on the classification. Each data distributed storage system has an independent fusion storage module to match and process fusion requests. This method can effectively cope with the distributed storage of continuous large-volume IoT data and requests for fusion analysis of data. The fusion analysis and data storage are carried out in a decentralized manner, thereby effectively improving data processing efficiency while reducing equipment load and requirements for equipment processing capabilities.

[0047] Preferably, according to the set or input data fusion strategy, the data fragments in the sub-database with fusion analysis requirements are assigned identifiers using the following method:

[0048] Each data slice contains multiple data packets;

[0049] The data fusion strategy includes a list of data fusion requirements in units of data packets;

[0050] If a data shard contains a data packet listed in the data fusion requirement list, the data shard is considered to be a data shard with fusion analysis requirements;

[0051] This method uses data packets as the basic unit, which is more in line with the characteristics of data in the Internet of Things. It also facilitates the smooth progress of subsequent fusion storage node matching operations, has high accuracy, and is not prone to errors.

[0052] Preferably, each fusion storage node stores one or more data fusion relationships using the following method:

[0053] The data fusion relationship corresponds to the fusion items in the data fusion requirement list;

[0054] Data packets that require data fusion in data shards are replaced with set characters;

[0055] Each data fusion relationship includes multiple set characters;

[0056] In data fusion, a data packet may be fused multiple times. This can be set and adjusted according to actual needs. Only the fusion relationship is stored, which requires less storage space and facilitates quick replacement and adjustment of the data fusion relationship.

[0057] Preferably, the fusion storage node establishes a connection with the child storage node storing the corresponding data shards by:

[0058] Multiple data shards in the same sub-database are distributed and stored in multiple sub-storage nodes;

[0059] With the set characters as matching items, multiple fusion storage nodes and multiple sub-storage nodes are connected.

[0060] Preferably, the data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion through the fusion storage nodes and feedback of the fused data using the following method:

[0061] The data distributed storage system receives external fusion access requests;

[0062] Decomposing the fusion access request into multiple data packets, matching the data fusion relationship according to the set characters corresponding to the data packets, and then matching the fusion storage node having the data fusion relationship;

[0063] Sending a fusion instruction to the fusion storage node;

[0064] The fusion storage node pulls the data shards from the corresponding sub-storage node according to the fusion instruction, and performs fusion analysis and processing according to the set rules, and the processed data is fed back to the outside world;

[0065] By entrusting the data fusion operation to the distributed fusion storage nodes, the computing load on the distributed storage system host can be greatly reduced and the data processing efficiency can be improved.

[0066] Preferably, the method further comprises the steps of:

[0067] The data shards that have been distributed and stored in the sub-database are deleted to cope with the infinite continuous data flow in the Internet of Things through limited data processing space.

[0068] Preferably, the classification rules for data classification include classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items; targeted adjustments can be made according to actual needs and there is no limitation on this.

[0069] A mid-point fusion system is applied to the above-mentioned Internet of Things data fusion method, such as Figure 2 See also Figure 3 , the system includes a data preprocessing module 1 and a distributed storage module 2;

[0070] The data preprocessing module 1 includes a buffer database 10, a data classification unit 11 and a data sharding unit 12; the distributed storage module 2 includes a plurality of data distributed storage systems 20;

[0071] The buffer database 10 is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit 11 is used to classify the data in the buffer database and store them in sub-databases with classification labels respectively; the data slicing unit 12 is used to perform slicing processing on the data in the current sub-database regularly or irregularly according to the set slicing size, and assign identifiers to the data slicing in the sub-database with fusion analysis requirements according to the set or input data fusion strategy;

[0072] A fusion storage module 200 is constructed in the data distributed storage system 20, and the fusion storage module 200 includes multiple fusion storage nodes 2000 and multiple sub-storage nodes 2001; each fusion storage node stores one or more data fusion relationships; one or more data slices are stored in the sub-storage node; the fusion storage node establishes a connection with the sub-storage node storing the corresponding data slice; when accessing, the data distributed storage system receives external fusion access requests, matches the fusion storage node, and performs data fusion and feedback of the fused data through the fusion storage node;

[0073] By using the system of the present application, large-volume IoT traffic is classified and processed in segments to avoid front-end data congestion, and multiple data distributed storage systems are constructed based on the classification. Each data distributed storage system has an independent fusion storage module to match and process fusion requests. This approach can effectively respond to the distributed storage of continuous large-volume IoT data and requests for fusion analysis of data. The fusion analysis and data storage are carried out in a decentralized manner, thereby effectively improving data processing efficiency while reducing equipment load and requirements for equipment processing capabilities.

[0074] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for fusion of Internet of Things data, characterized in that: The method comprises the following steps: Real-time acquisition of data to be stored in the Internet of Things is temporarily stored in the buffer database; data in the buffer database is classified and stored in sub-databases with classification labels; Perform sharding processing on the data in the current sub-database regularly or irregularly according to the set sharding size, and assign identifiers to the data shards in the sub-database that have fusion analysis requirements according to the set or input data fusion strategy; Constructing a data distributed storage system corresponding to each sub-database, constructing a fusion storage module in the data distributed storage system, wherein the fusion storage module includes a plurality of fusion storage nodes and a plurality of sub-storage nodes; Each fusion storage node stores one or more data fusion relationships; each child storage node stores one or more data shards; The fusion storage node establishes a connection with the child storage node storing the corresponding data shards; During access, the data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion and feedback of fusion data through fusion storage nodes.

2. The method for fusion of Internet of Things data according to claim 1, characterized in that: The method of assigning identifiers to data segments in the sub-database that have fusion analysis requirements according to the set or input data fusion strategy adopts: Each data slice contains multiple data packets; The data fusion strategy includes a list of data fusion requirements in units of data packets; If a data packet listed in the data fusion requirement list exists in a data slice, the data slice is considered to be a data slice with fusion analysis requirements.

3. The method for fusion of Internet of Things data according to claim 2, characterized in that: Each fusion storage node stores one or more data fusion relationships using the following method: The data fusion relationship corresponds to the fusion items in the data fusion requirement list; Data packets that require data fusion in data shards are replaced with set characters; Each data fusion relationship includes a plurality of setting characters.

4. The method for fusion of Internet of Things data according to claim 3, characterized in that: The method for establishing a connection between the fusion storage node and the child storage node storing the corresponding data slice is as follows: Multiple data shards in the same sub-database are distributed and stored in multiple sub-storage nodes; With the set characters as matching items, multiple fusion storage nodes and multiple sub-storage nodes are connected.

5. The method for fusion of Internet of Things data according to claim 4, characterized in that: The data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion through the fusion storage nodes and feedback of the fused data using the following method: The data distributed storage system receives external fusion access requests; Decomposing the fusion access request into multiple data packets, matching the data fusion relationship according to the set characters corresponding to the data packets, and then matching the fusion storage node having the data fusion relationship; Sending a fusion instruction to the fusion storage node; The fusion storage node pulls the data shards from the corresponding sub-storage node according to the fusion instruction, and performs fusion analysis and processing according to the set rules, and the processed data is fed back to the outside world.

6. The method for fusion of Internet of Things data according to claim 1, characterized in that: The method further comprises the steps of: Delete the data shards that have been distributed stored in the sub-database.

7. The method for fusion of Internet of Things data according to claim 1, characterized in that: The classification rules of the data classification include classification by data generation or modification time period, classification by sensor type, classification by devices in the Internet of Things, and classification by custom items.

8. A middle-point fusion system, applied to the Internet of Things data fusion method according to any one of claims 1 to 7, characterized in that: The system includes a data preprocessing module and a distributed storage module; The data preprocessing module includes a buffer database, a data classification unit and a data sharding unit; the distributed storage module includes a plurality of data distributed storage systems; The buffer database is used to obtain the data to be stored in the Internet of Things in real time and temporarily store it; the data classification unit is used to classify the data in the buffer database and store them in sub-databases with classification labels respectively; the data slicing unit is used to slice the data in the current sub-database regularly or irregularly according to the set slice size, and assign identifiers to the data slices in the sub-database that have fusion analysis requirements according to the set or input data fusion strategy; A fusion storage module is constructed in the data distributed storage system, and the fusion storage module includes multiple fusion storage nodes and multiple sub-storage nodes; each fusion storage node stores one or more data fusion relationships; and the sub-storage node stores one or more data fragments; The fusion storage node establishes a connection with the child storage node storing the corresponding data shards; During access, the data distributed storage system receives external fusion access requests, matches fusion storage nodes, and performs data fusion and feedback of fusion data through fusion storage nodes.

Images