Data fragmentation method applied to Internet of Things system and mid-division fusion system

Abstract

The invention relates to a data fragmentation method and a middle-division fusion system applied to an Internet of Things system, and the method comprises the following steps: pulling and temporarily storing sensor data in the Internet of Things in real time, carrying out the data classification, and moving the data to a corresponding classification queue; constructing a mapping subunit for each classification queue, wherein the mapping subunits are used for copying data segments with fixed positions and fixed lengths in the classification queues according to a set time interval; carrying out fusion processing on the multiple data segments, deleting repeated data, and carrying out data fragmentation on the fused data according to a set fragmentation rule; aiming at the characteristics of real-time performance and continuity of sensor data in the Internet of Things, a unique classification queue temporary storage mode is designed, and a mapping subunit is adopted to carry out segmented copying of the data and a later fusion fragmentation method, so that real-time data is converted into fixed data, fragmentation processing can be carried out by adopting an existing fragmentation algorithm, and therefore, the real-time data can be converted into fixed data. And the system development period and the development cost are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data distributed storage, and more specifically, to a data sharding method and a data fusion system applied to an Internet of Things system. Background Art

[0002] The Internet of Things originated in the field of media and is the third revolution of the information technology industry. The Internet of Things refers to connecting any object to the network through information sensing devices according to agreed protocols. Objects exchange and communicate information through information transmission media to achieve intelligent identification, positioning, tracking, supervision and other functions; in the Internet of Things, sensors play a very important role and are the core link to achieve interconnection; existing conventional Internet of Things sensor data are mostly stored locally, which is limited by local memory resources, and storage often cannot meet demand, and it also brings great restrictions to the expansion of the Internet of Things;

[0003] In order to realize the distributed storage of sensor data in the Internet of Things, real-time data sharding processing is one of the important links. However, since the real-time amount of sensor data is usually large in scale and there is uncertainty in continuity (unbalanced data flow rate), traditional distributed storage sharding methods are mostly aimed at static data of fixed capacity. There is currently no real-time sharding method for applications in this field. A data sharding method and a data fusion system applied to the Internet of Things system are needed to fill the gap in the industry and reduce the high cost of redeveloping algorithms. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data sharding method applied to an Internet of Things system and a data fusion system in view of the above-mentioned defects of the prior art.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A data sharding method for an Internet of Things system is constructed, which includes the following steps:

[0007] Step 1: Pull sensor data from the Internet of Things in real time and temporarily store the sensor data in a temporary database;

[0008] Step 2: Classify the sensor data in the temporary database in real time according to the set classification strategy and move it to the corresponding classification queue;

[0009] Step 3: When new data is added to each classification queue, the data in the classification queue is shifted back by the corresponding number of bits, and the end of the classification queue deletes the data according to the set speed;

[0010] Step 4: construct a mapping subunit for each classification queue, the mapping subunit is used to copy a data segment of a fixed position and a fixed length in the classification queue according to a set time interval;

[0011] When the mapping subunit is set to a data segment with a fixed position and a fixed length in a set time interval and a classification queue, it is necessary to first count the maximum and minimum values ​​of the data flow speed in the queue, and set the time interval, position and length based on the maximum value to ensure that no packets are missed;

[0012] Step 5: Fuse the continuous multiple data segments obtained by the mapping sub-unit, delete duplicate data, and fragment the merged data according to the set fragmentation rules.

[0013] The data sharding method applied to the Internet of Things system of the present invention, wherein the step five also includes the method:

[0014] The remaining data after each fusion sharding process will participate in the next data fusion process.

[0015] The data sharding method applied to the Internet of Things system of the present invention, wherein the real-time pulling of sensor data in the Internet of Things and the temporary storage of the sensor data in the temporary storage database adopt the method:

[0016] Build a data processing server and establish a communication connection between the data processing server and the Internet of Things platform;

[0017] One or more levels of temporary storage databases are configured in the data processing server;

[0018] Receive sensor data uploaded by the IoT platform in real time and store it temporarily in the temporary database.

[0019] The data sharding method applied to the Internet of Things system of the present invention, wherein the sensor data in the temporary database is classified in real time according to the set classification strategy and moved to the corresponding classification queue using the method:

[0020] Filter and analyze the sensor data in the temporary database to determine the type and time attributes of each data packet in the sensor data;

[0021] Perform real-time data classification based on the set classification strategy;

[0022] Match the classified data with the classification queue. If the match is successful, the data is moved into the classification queue. If the match fails, feedback is given to the outside.

[0023] Generate a new classification queue for the data that fails to match.

[0024] The data sharding method applied to the Internet of Things system of the present invention, wherein the setting classification strategy includes classification based on time or classification based on sensor type;

[0025] Based on time division, data can be classified according to the data collection time; based on sensor type division, data of similar types can be divided into the same area.

[0026] The data slicing method applied to the Internet of Things system of the present invention, wherein the mapping subunit is constructed for each classification queue, and the mapping subunit is used to copy the data segments of fixed position and fixed length in the classification queue according to the set time interval by:

[0027] A mapping database is configured, wherein the mapping database includes a plurality of mapping subunits corresponding to the classification queues one by one;

[0028] Establishing a mapping relationship between the mapping subunit and a fixed position and a fixed length area of ​​the corresponding classification queue;

[0029] Copy the data segments with fixed positions and fixed lengths in the classification queue at set time intervals.

[0030] The data sharding method applied to the Internet of Things system of the present invention, wherein the continuous multiple data segments acquired by the mapping subunit are fused, duplicate data are deleted, and the fused data are sharded according to the set sharding rules using:

[0031] Each time the data segment acquired by the mapping subunit is transferred and stored in the set fusion space;

[0032] When the number of transfers reaches the set number, the fusion space is processed for data fusion and the repeated parts in adjacent data segments are deleted;

[0033] The fused data is sent to the distributed storage server for data sharding processing.

[0034] A data sharding and fusion system, using the data sharding method applied to the Internet of Things system as described above, wherein the system includes a data pulling unit, a temporary storage database, a mapping database and a data sharding unit;

[0035] The data pulling unit is used to pull sensor data in the Internet of Things in real time and temporarily store the sensor data in a temporary storage database;

[0036] The temporary database performs real-time data classification on the sensor data in the temporary database according to the set classification strategy, and moves the data to the corresponding classification queue; when new data is added to each classification queue, the data in the classification queue is shifted back by the corresponding number of bits, and the end of the classification queue deletes the data according to the set speed;

[0037] The mapping database includes a plurality of mapping subunits corresponding to the classification queues one by one, and the mapping subunits are used to copy data segments of fixed position length in the classification queues according to set time intervals;

[0038] The data slicing unit is used to fuse the continuous multiple data segments obtained by the mapping subunit, delete duplicate data, and slice the fused data according to a set size, and the redundant data participates in the next data fusion process.

[0039] The beneficial effects of the present invention are as follows: by applying the method of the present application, a unique classification queue temporary storage method is designed in view of the real-time and continuity characteristics of sensor data in the Internet of Things, and a mapping subunit is used to perform segmented replication of data and a method of later fusion sharding, so that the real-time acquisition and release of sensor data and data sharding are processed by two completely independent channels without interfering with each other, thereby realizing the conversion of real-time data into fixed data, and sharding processing can be performed using the existing sharding algorithm, thereby reducing the system development cycle and development cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:

[0041] Figure 1 This is a flow chart of a data sharding method applied to an Internet of Things system according to a preferred embodiment of the present invention;

[0042] Figure 2 It is a principle block diagram of the center-separation fusion system of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0044] The data slicing method applied to the Internet of Things system in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0045] S01: Pull sensor data from the Internet of Things in real time and temporarily store the sensor data in a temporary database;

[0046] S02: classify the sensor data in the temporary database in real time according to the set classification strategy, and move it to the corresponding classification queue;

[0047] S03: When new data is added to each classification queue, the data in the classification queue is shifted back by the corresponding number of bits, and the end of the classification queue deletes the data according to the set speed;

[0048] S04: constructing a mapping subunit for each classification queue, the mapping subunit being used to copy a data segment of a fixed position and a fixed length in the classification queue according to a set time interval;

[0049] S05: performing fusion processing on the continuous multiple data segments obtained by the mapping sub-unit, deleting duplicate data, and performing data sharding on the fused data according to the set sharding rules.

[0050] By applying the method and approach of the present application, a unique classification queue temporary storage method is designed for the real-time and continuity characteristics of sensor data in the Internet of Things, and a mapping subunit is used to perform segmented replication of data and a method of later fusion sharding, so that the real-time acquisition and release of sensor data and data sharding are processed by two completely independent channels without interfering with each other, and the real-time data is converted into fixed data. The existing sharding algorithm can be used for sharding processing, which reduces the system development cycle and development cost;

[0051] It should be noted that when the mapping subunit is set to set a data segment with a fixed position and a fixed length in a classification queue according to a set time interval, it is necessary to first count the maximum and minimum values ​​of the data flow speed in the queue, and set the time interval, position and length based on the maximum value to ensure that no packets are missed. After the setting is completed, the setting can be maintained for processing without changing it again;

[0052] Preferably, step five also includes a method:

[0053] The remaining data after each fusion and sharding process participates in the next data fusion process; this processing method can effectively solve the problem of inconsistent packet sizes each time obtained due to data duplication.

[0054] Preferably, the sensor data in the Internet of Things is pulled in real time, and the sensor data is temporarily stored in a temporary database using the following method:

[0055] Build a data processing server and establish a communication connection between the data processing server and the Internet of Things platform;

[0056] One or more levels of temporary storage databases are configured in the data processing server;

[0057] Receive sensor data uploaded by the IoT platform in real time and store it temporarily in the temporary database;

[0058] Preferably, the sensor data in the temporary database is classified in real time according to the set classification strategy, and moved to the corresponding classification queue using the method:

[0059] Filter and analyze the sensor data in the temporary database to determine the type and time attributes of each data packet in the sensor data;

[0060] Perform real-time data classification based on the set classification strategy;

[0061] Match the classified data with the classification queue. If the match is successful, the data is moved into the classification queue. If the match fails, feedback is given to the outside.

[0062] Generate a new classification queue for the data that failed to match;

[0063] To maintain the scalability of the data, each time a new classification queue is generated, a corresponding mapping subunit needs to be generated.

[0064] Preferably, setting the classification strategy includes dividing based on time or dividing based on sensor type;

[0065] Based on time division, data can be classified according to the data collection time; based on sensor type division, data of similar types can be divided into the same area.

[0066] Preferably, a mapping subunit is constructed for each classification queue, and the mapping subunit is used to copy a data segment of a fixed position and a fixed length in the classification queue according to a set time interval by:

[0067] A mapping database is configured, wherein the mapping database includes a plurality of mapping subunits corresponding to the classification queues one by one;

[0068] Establishing a mapping relationship between the mapping subunit and a fixed position and a fixed length area of ​​the corresponding classification queue;

[0069] Copy the data segments with fixed positions and fixed lengths in the classification queue at set time intervals.

[0070] Preferably, the continuous multiple data segments acquired by the mapping subunit are merged, duplicate data is deleted, and the merged data is fragmented according to the set fragmentation rules using:

[0071] Each time the data segment acquired by the mapping subunit is transferred and stored in the set fusion space;

[0072] When the number of transfers reaches the set number, the fusion space is processed for data fusion and the repeated parts in adjacent data segments are deleted;

[0073] The fused data is sent to the distributed storage server for data sharding processing.

[0074] A data slicing and fusion system, using the data slicing method applied to the Internet of Things system as described above, such as Figure 2 As shown, the system includes a data pulling unit 100, a temporary storage database 101, a mapping database 102 and a data sharding unit 103;

[0075] The data pulling unit 100 is used to pull sensor data in the Internet of Things in real time and temporarily store the sensor data in a temporary storage database;

[0076] Temporary database 101, classifies sensor data in the temporary database in real time according to the set classification strategy, and moves it to the corresponding classification queue 1010; when new data is added to each classification queue 1010, the data in the classification queue is shifted back by the corresponding number of bits, and the end of the classification queue deletes data according to the set speed;

[0077] The mapping database 102 includes a plurality of mapping subunits 1020 corresponding to the classification queues one by one, and the mapping subunits are used to copy data segments of fixed position length in the classification queues according to a set time interval;

[0078] The data slicing unit 103 is used to fuse the continuous multiple data segments obtained by the mapping sub-unit, delete duplicate data, and slice the fused data according to a set size, and the redundant data participates in the next data fusion process.

[0079] The system using the present application designs a unique classified queue temporary storage method based on the real-time and continuity characteristics of sensor data in the Internet of Things, and adopts a mapping sub-unit to perform segmented replication of data and a method of later fusion sharding, so that the real-time acquisition and release of sensor data and data sharding are processed by two completely independent channels without interfering with each other, thereby realizing the conversion of real-time data into fixed data. The existing sharding algorithm can be used for sharding processing, which reduces the system development cycle and development cost.

[0080] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A data sharding method applied to an Internet of Things system, characterized in that: The following steps are involved: Step 1: Pull sensor data from the Internet of Things in real time and temporarily store the sensor data in a temporary database; Step 2: Classify the sensor data in the temporary database in real time according to the set classification strategy and move it to the corresponding classification queue; Step 3: When new data is added to each classification queue, the data in the classification queue is shifted back by the corresponding number of bits, and the end of the classification queue deletes the data according to the set speed; Step 4: construct a mapping subunit for each classification queue, the mapping subunit is used to copy a data segment of a fixed position and a fixed length in the classification queue according to a set time interval; When the mapping subunit is set to a data segment with a fixed position and a fixed length in a set time interval and a classification queue, it is necessary to first count the maximum and minimum values ​​of the data flow speed in the queue, and set the time interval, position and length based on the maximum value to ensure that no packets are missed; Step 5: Fuse the continuous multiple data segments obtained by the mapping sub-unit, delete duplicate data, and fragment the merged data according to the set fragmentation rules.

2. The data sharding method applied to the Internet of Things system according to claim 1 is characterized in that: The step five also includes a method: The remaining data after each fusion sharding process will participate in the next data fusion process.

3. The data sharding method applied to the Internet of Things system according to claim 1 is characterized in that: The method for pulling sensor data in the Internet of Things in real time and temporarily storing the sensor data in a temporary database adopts: Build a data processing server and establish a communication connection between the data processing server and the Internet of Things platform; One or more levels of temporary storage databases are configured in the data processing server; Receive sensor data uploaded by the IoT platform in real time and store it temporarily in the temporary database.

4. The data sharding method applied to the Internet of Things system according to claim 3 is characterized in that: The method of classifying the sensor data in the temporary database in real time according to the set classification strategy and moving it to the corresponding classification queue is as follows: Filter and analyze the sensor data in the temporary database to determine the type and time attributes of each data packet in the sensor data; Perform real-time data classification based on the set classification strategy; Match the classified data with the classification queue. If the match is successful, the data is moved into the classification queue. If the match fails, feedback is given to the outside. Generate a new classification queue for the data that fails to match.

5. The data sharding method applied to the Internet of Things system according to claim 4 is characterized in that: The setting of the classification strategy includes classification based on time or classification based on sensor type; Based on time division, data can be classified according to the data collection time; based on sensor type division, data of similar types can be divided into the same area.

6. The data sharding method applied to the Internet of Things system according to claim 4 is characterized in that: The mapping subunit is constructed for each classification queue, and the mapping subunit is used to copy the data segments of fixed position and fixed length in the classification queue according to the set time interval by: A mapping database is configured, wherein the mapping database includes a plurality of mapping subunits corresponding to the classification queues one by one; Establishing a mapping relationship between the mapping subunit and a fixed position and a fixed length area of ​​the corresponding classification queue; Copy the data segments with fixed positions and fixed lengths in the classification queue at set time intervals.

7. The data sharding method applied to the Internet of Things system according to claim 6 is characterized in that: The continuous multiple data segments obtained by the mapping sub-unit are merged, duplicate data is deleted, and the merged data is fragmented according to the set fragmentation rules: Each time the data segment acquired by the mapping subunit is transferred and stored in the set fusion space; When the number of transfers reaches the set number, the fusion space is processed for data fusion and the repeated parts in adjacent data segments are deleted; The fused data is sent to the distributed storage server for data sharding processing.

8. A data sharding and fusion system, using the data sharding method for an Internet of Things system as claimed in any one of claims 1 to 7, characterized in that: The system includes a data pulling unit, a temporary storage database, a mapping database and a data sharding unit; The data pulling unit is used to pull sensor data in the Internet of Things in real time and temporarily store the sensor data in a temporary storage database; The temporary database performs real-time data classification on the sensor data in the temporary database according to the set classification strategy, and moves the data to the corresponding classification queue; when new data is added to each classification queue, the data in the classification queue is shifted back by the corresponding number of bits, and the end of the classification queue deletes the data according to the set speed; The mapping database includes a plurality of mapping subunits corresponding to the classification queues one by one, and the mapping subunits are used to copy data segments of fixed position length in the classification queues according to set time intervals; The data slicing unit is used to fuse the continuous multiple data segments obtained by the mapping subunit, delete duplicate data, and slice the fused data according to a set size, and the redundant data participates in the next data fusion process.

Images