A data transmission system, method and medium based on the Internet of Things and big data

Abstract

The embodiment of the present invention discloses a data transmission system, method and medium based on the Internet of Things and big data. The system includes: a collection unit, which is used for diverse data collection; a preprocessing unit, which is used to package the data collected by the collection unit according to a preset period to form a data packet, and process the data in each data packet to obtain a corresponding data change rate and data correlation index; a marking unit, which is used to compare the obtained data change rate and data correlation index with a preset threshold value, and mark the data packet corresponding to the data exceeding the threshold value; a transmission unit, which is used to give priority to the transmission of the marked data packet; the beneficial effect is: by acquiring the data change rate and data correlation index in each collected data packet, marking the data packet exceeding the threshold value, so that data with larger differences are transmitted first, which is beneficial to subsequent data processing and improves the performance of data fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data transmission system, method and medium based on the Internet of Things and big data. Background Art

[0002] In the big data application system, the front end is connected to various sensors and collection devices deployed on the Internet of Things, and then the collected data is transmitted to the processing center through the communication link, so that the subsequent system can perform data fusion, association and mining. However, due to the huge amount of data, it also brings pressure on data transmission. At present, the existing technologies all consider how to improve the transmission efficiency, and do not consider the transmission based on the characteristics of the data to be transmitted itself, and the perspective of facilitating the fusion of subsequent data. For example, a large amount of data of the same type is transmitted, which results in the defect of low data fusion performance in the subsequent data processing process. Summary of the invention

[0003] In view of the technical defects in the prior art, the purpose of the embodiments of the present invention is to provide a data transmission system, method and medium based on the Internet of Things and big data, so as to overcome the defect of low data fusion performance in the prior art.

[0004] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a data transmission system based on the Internet of Things and big data, the system comprising:

[0005] A collection unit, used for collecting diverse data; wherein the collected data has time and correlation characteristics;

[0006] A preprocessing unit, used for packaging the data collected by the collection unit according to a preset period to form data packets, and processing the data in each data packet to obtain a corresponding data change rate and data correlation index;

[0007] A marking unit, used to compare the obtained data change rate and data correlation index with a preset threshold value, and mark the data packets corresponding to the data exceeding the threshold value;

[0008] Transmission unit for:

[0009] Prioritize transmission of marked data packets;

[0010] And compress the transmitted data packets before transmitting them.

[0011] Preferably, the preprocessing unit is further used for: dividing the collected data into real-time data and non-real-time data.

[0012] Preferably, the preprocessing unit is further used to: calculate the similarity of data between each data packet; at the same time, the marking unit also marks the data packets with low similarity.

[0013] Preferably, the transmission unit is further used for:

[0014] Set the transmission time for each data packet separately;

[0015] During transmission, if the transmission time is exceeded, the corresponding data packet will be retained and sent again when the link is idle.

[0016] In a second aspect, an embodiment of the present invention further provides a data transmission method based on the Internet of Things and big data, which is applied to a data transmission system based on the Internet of Things and big data described in the first aspect, and the method includes:

[0017] Conducting diversified data collection; wherein the collected data has time and correlation characteristics;

[0018] Packing the collected data according to a preset period to form data packets, and processing the data in each data packet to obtain corresponding data change rate and data correlation indicators;

[0019] Comparing the obtained data change rate and data correlation index with a preset threshold, and marking data packets corresponding to data exceeding the threshold;

[0020] Prioritize transmission of marked data packets;

[0021] And compress the transmitted data packets before transmitting them.

[0022] Preferably, the method further comprises:

[0023] Divide the collected data into real-time data and non-real-time data;

[0024] The transmission link includes a real-time link and a non-real-time link, and the real-time data is transmitted through the real-time link, and the non-real-time data is transmitted through the non-real-time link.

[0025] Preferably, the method further comprises:

[0026] Calculate the similarity of data between each data packet;

[0027] And mark the data packets with low similarity.

[0028] Preferably, during transmission, a transmission time is set for each data packet;

[0029] During transmission, if the transmission time is exceeded, the corresponding data packet will be retained and sent again when the link is idle.

[0030] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the second aspect are implemented.

[0031] By implementing the embodiment of the present invention, the data change rate and data correlation index in each collected data packet are obtained, and the data packets exceeding the threshold are marked, so as to obtain the data difference, so that the data with larger difference is transmitted first, which is beneficial to the subsequent data processing and improves the performance of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0033] Figure 1 It is a structural diagram of a data transmission system based on the Internet of Things and big data provided by an embodiment of the present invention;

[0034] Figure 2 It is a flow chart of a data transmission method based on the Internet of Things and big data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or subcombination. In addition, it should be understood by those of ordinary skill in the art that the figures provided herein are for illustrative purposes and that the figures are not necessarily drawn to scale.

[0037] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0038] Please refer to Figure 1 , an embodiment of the present invention provides a data transmission system based on the Internet of Things and big data, the system comprising:

[0039] The collection unit is used for collecting diverse data; wherein the collected data has time and correlation characteristics.

[0040] Specifically, the collection unit forms an Internet of Things through multiple collection nodes to measure and generate diverse data of different types and forms, such as analog, digital, picture, sound and video, etc. At the same time, data is also stored; wherein each collection node includes a sensor, a controller and an Internet of Things module; the correlation characteristics include the associated equipment, location and whether there is a correlation influence with other data.

[0041] The preprocessing unit is used to package the data collected by the collection unit according to a preset period to form data packets, and process the data in each data packet to obtain a corresponding data change rate and data correlation index.

[0042] Specifically, in the subsequent fusion processing of IoT data, such as model training and parameter adjustment, a large amount of data of different types and characteristics is required. Therefore, the greater the data change rate in the data packet and the smaller the data correlation, the more conducive it is to the subsequent data fusion association and mining, and it is necessary to ensure its priority transmission. The data in the data packet is not a single type or category of data.

[0043] The marking unit is used to compare the obtained data change rate and data correlation index with a preset threshold value, and mark the data packets corresponding to the data exceeding the threshold value.

[0044] Specifically, the numbers corresponding to the thresholds can be flexibly set. By combining the data change rate and data correlation indicators, data packets with large differences can be marked more accurately, thereby ensuring their priority transmission, making subsequent data processing more timely and accurate.

[0045] Furthermore, the data in the data packet may be further marked for packetization to obtain corresponding sub-data packets for priority transmission; for example, data with a longer sampling period, new sampled data, etc.

[0046] In another embodiment, to further reflect the differences between the data packets, the value of the data packets for data fusion is obtained;

[0047] The preprocessing unit is also used to calculate the similarity of data between each data packet; at the same time, the marking unit also marks the data packets with low similarity.

[0048] Transmission unit for:

[0049] Prioritize transmission of marked data packets;

[0050] And compress the transmitted data packets before transmitting them.

[0051] Specifically, during implementation, the rate of file transmission is further increased through compression processing; of course, data transmission performance can also be improved through parallel transmission, data encoding and other technologies.

[0052] Furthermore, in order to achieve better transmission priority during data transmission, the preprocessing unit is also used to: divide the collected data into real-time data and non-real-time data;

[0053] Correspondingly, the transmission link includes a real-time link and a non-real-time link, the real-time data is transmitted through the real-time link, and the non-real-time data is transmitted through the non-real-time link.

[0054] The above technical solution obtains the data change rate and data correlation indicators in each collected data packet, marks the data packets that exceed the threshold, and thus derives the data difference, so that data with larger differences are transmitted first, which is beneficial to subsequent data processing and improves the performance of data fusion.

[0055] Furthermore, in order to improve the reliability of data transmission, the transmission unit is also used to:

[0056] Set the transmission time for each data packet separately;

[0057] During transmission, if the transmission time is exceeded, the corresponding data packet will be retained and sent again when the link is idle.

[0058] The transmission time can be manually set or dynamically and flexibly set according to the size of the data packet, which will not be described in detail here;

[0059] When a network failure or transmission failure occurs, data recovery can be performed based on the aforementioned stored data, and transmission can be continued or restarted from where the data was interrupted, thereby improving the reliability of transmission.

[0060] Furthermore, in another embodiment, based on the above scheme, during data transmission, the selected transmission link is selected based on the principle of path reliability, wherein the path reliability means that the routing node has at least two optional transmission links. When the transmission is successful, the remaining transmission links are released for transmission of other nodes to improve utilization. This can also avoid the defect in the prior art that a node fails and transmission cannot be made due to the shortest path selection method.

[0061] Based on the same inventive concept, an embodiment of the present invention further provides a data transmission method based on the Internet of Things and big data, which is applied to a data transmission system based on the Internet of Things and big data as described above. Figure 2 As shown, the method includes:

[0062] S101, performing diverse data collection, wherein the collected data has time and correlation characteristics;

[0063] S102, packaging the collected data according to a preset period to form data packets, and processing the data in each data packet to obtain a corresponding data change rate and data correlation index;

[0064] S103, comparing the obtained data change rate and data correlation index with a preset threshold, and marking data packets corresponding to data exceeding the threshold;

[0065] S104, preferentially transmitting the marked data packets;

[0066] And compress the transmitted data packets before transmitting them.

[0067] When applied, the method further comprises:

[0068] Divide the collected data into real-time data and non-real-time data;

[0069] The transmission link includes a real-time link and a non-real-time link, and the real-time data is transmitted through the real-time link, and the non-real-time data is transmitted through the non-real-time link.

[0070] At the same time, in order to better reflect the differences between the data packets, the method further includes:

[0071] Calculate the similarity of data between each data packet;

[0072] And mark the data packets with low similarity.

[0073] Furthermore, during transmission, a transmission time is set for each data packet;

[0074] During transmission, if the transmission time is exceeded, the corresponding data packet will be retained and sent again when the link is idle.

[0075] It should be noted that for more specific work processes in the method embodiments, please refer to the aforementioned system embodiment section, which will not be repeated here.

[0076] This solution obtains the data change rate and data correlation indicators in each collected data packet, marks the data packets that exceed the threshold, and thus derives the data difference, so that data with larger differences are transmitted first, which is beneficial to subsequent data processing and improves the performance of data fusion.

[0077] In this embodiment, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of an embodiment of a data transmission method based on the Internet of Things and big data.

[0078] Specifically, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0079] The code of the computer program may be in source code form, object code form, executable file or some intermediate form.

[0080] Computer-readable storage media may include cache, high-speed random access memory (RAM), such as the common double data rate synchronous dynamic random access memory (DDR SDRAM), and may also include non-volatile memory (NVRAM), such as one or more read-only memories (ROMs), disk storage devices, flash memory devices, or other non-volatile solid-state memory devices such as optical disks (CD-ROMs, DVD-ROMs), floppy disks or data tapes, etc.

[0081] Those of ordinary skill in the art will appreciate that the modules and steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0083] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A data transmission system based on the Internet of Things and big data, characterized in that: The system comprises: A collection unit, used for collecting diverse data; wherein the collected data has time and correlation characteristics; the correlation characteristics include the associated equipment, location and whether there is a correlation influence with other data; A preprocessing unit, used for packaging the data collected by the collection unit according to a preset period to form data packets, and processing the data in each data packet to obtain a corresponding data change rate and data correlation index; A marking unit, used to compare the obtained data change rate and data correlation index with a preset threshold value, and mark the data packets corresponding to the data exceeding the threshold value; mark the data packets with large differences, thereby ensuring their priority transmission; Transmission unit for: Prioritize transmission of marked data packets; and compressing the transmitted data packets before transmitting them; The preprocessing unit is further used to: divide the collected data into real-time data and non-real-time data; the transmission link includes a real-time link and a non-real-time link, the real-time data is transmitted through the real-time link, and the non-real-time data is transmitted through the non-real-time link; During data transmission, the selected transmission link is selected based on the principle of path reliability, where the path reliability means that the routing node has at least two optional transmission links. When the transmission is successful, the remaining transmission links are released for transmission of other nodes to improve the utilization rate; Further marking the data in the data packet for subpacketization to obtain corresponding sub-data packets for priority transmission; The transmission unit is also used for: Set the transmission time for each data packet separately; During transmission, if the transmission time is exceeded, the corresponding data packet will be retained and sent again when the link is idle.

2. A data transmission system based on the Internet of Things and big data as claimed in claim 1, characterized in that: The preprocessing unit is also used to calculate the similarity of data between each data packet; at the same time, the marking unit also marks the data packets with low similarity.

3. A data transmission method based on the Internet of Things and big data, characterized in that: The method applied to the data transmission system based on the Internet of Things and big data as described in claim 1 comprises: Perform diverse data collection; wherein the collected data has time and correlation characteristics; the correlation characteristics include the associated equipment, location and whether there is a correlation influence with other data; Packing the collected data according to a preset period to form data packets, and processing the data in each data packet to obtain corresponding data change rate and data correlation indicators; Compare the obtained data change rate and data correlation index with a preset threshold, and mark the data packets corresponding to the data exceeding the threshold; mark the data packets with greater differences, thereby ensuring their priority transmission; Prioritize transmission of marked data packets; and compressing the transmitted data packets before transmitting them; The method further comprises: Divide the collected data into real-time data and non-real-time data; The transmission link includes a real-time link and a non-real-time link, and the real-time data is transmitted through the real-time link, and the non-real-time data is transmitted through the non-real-time link; The transmission link includes a real-time link and a non-real-time link, the real-time data is transmitted through the real-time link, and the non-real-time data is transmitted through the non-real-time link; During data transmission, the selected transmission link is selected based on the principle of path reliability, where the path reliability means that the routing node has at least two optional transmission links. When the transmission is successful, the remaining transmission links are released for transmission of other nodes to improve the utilization rate; Further marking the data in the data packet for subpacketization to obtain corresponding sub-data packets for priority transmission; When transmitting, set the transmission time for each data packet separately; During transmission, if the transmission time is exceeded, the corresponding data packet will be retained and sent again when the link is idle.

4. A data transmission method based on the Internet of Things and big data as claimed in claim 3, characterized in that: The method further comprises: Calculate the similarity of data between each data packet; And mark the data packets with low similarity.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a data transmission method based on the Internet of Things and big data as described in any one of claims 3 to 4 are implemented.

Images