Data acquisition system based on Internet of Things equipment

Abstract

The invention discloses a data acquisition system based on Internet of Things equipment, which comprises an Internet of Things device arranged at the front end, a gateway in communication connection with the Internet of Things device, and a data acquisition unit in communication connection with the gateway, the exception management module is used for identifying the exception condition of data transmission in the working process, and the exception management module comprises a time period window sub-module which is used for setting a time period window T; the statistical sub-module is used for carrying out periodic statistics on the data receiving quantity by calling a time period window T; the storage sub-module is used for storing the data receiving quantity counted in different time period windows T; and the judgment submodule is used for comparing the total data receiving amount of the current time period window T with the total data receiving amount of the previous time period window T and outputting a judgment result according to a comparison result.

Description

Technical Field

[0001] The present invention belongs to the technical field of data communication, and in particular to a data acquisition system based on Internet of Things devices. Background Art

[0002] The Internet of Things is a data transmission and processing network formed by the organic combination of various information sensing devices. The front end of the Internet of Things is often various data collection and acquisition devices. By acquiring various data generated in production and operation activities, the back end will process and separate them to achieve the purpose of reasonable, scientific, accurate and efficient management. It is especially widely used in industrial applications.

[0003] In industry, various data acquisition devices or systems are set up to obtain different data at all times, but electronic equipment has a service life and will inevitably be accompanied by maintenance or even replacement. On the other hand, industrial development is also transformative, so there will also be new demands for data acquisition devices.

[0004] It is relatively easy to simply repair, replace or add these data acquisition devices at the front end. However, for the platform that collects data and performs processing and analysis at the back end, the amount of data processed is too large and the sensitivity to the state changes of these data acquisition devices at the front end is very low, resulting in the platform that collects data and performs processing and analysis at the back end being unable to accurately obtain data information. Summary of the invention

[0005] In view of the above technical problems, this application proposes a data collection system based on IoT devices. The specific technical solutions are as follows:

[0006] The data acquisition system based on the Internet of Things device includes an Internet of Things device arranged at the front end, a gateway connected to the Internet of Things device, and a data collector connected to the gateway. The gateway is provided with an abnormal management module, and the abnormal management module is used to identify abnormal situations of data transmission during the working process. The abnormal management module includes:

[0007] A time period window submodule, wherein the time period window submodule is used to set a time period window T;

[0008] A statistical submodule, wherein the statistical submodule performs statistics on the amount of periodic data received by calling a time period window T;

[0009] A storage submodule, the storage submodule is used to store the amount of data received statistically in different time period windows T;

[0010] The judgment submodule obtains the total amount of data received in the current time period window T and the total amount of data received in the previous time period window T, compares them, and outputs a judgment result according to the comparison result.

[0011] Furthermore, the abnormal management module identifies abnormal data transmission through strategy 1, and the strategy 1 is:

[0012] (1) Set a time period window T;

[0013] (2) Continuously counting the amount of data received based on the time period window T to obtain the total amount of data received S1 within the time period window T;

[0014] (3) Retrieve the total amount of data received S2 counted in the previous time period window T, and compare the total amount of data received S1 in the current time period window T with the total amount of data received S2 in the previous time period window T, and output a judgment result based on the comparison result:

[0015] If S1 is greater than S2, the abnormal transmission is judged to be a new IoT device;

[0016] If S1 is less than S2, the abnormal transmission is judged to be due to the suspension of use or damage of the IoT device;

[0017] If S1 is equal to S2, it is determined that there is no abnormal transmission.

[0018] Furthermore, the gateway is also provided with an exception handling module, which is enabled when the exception management module identifies abnormal data transmission, and is used to locate the IoT device where abnormal transmission occurs. The exception handling module includes:

[0019] A filtering submodule, wherein the filtering submodule can call the routing table and the ARP cache table in the gateway, thereby filling the physical addresses stored therein into the filtering layer;

[0020] An address pool, the address pool is used to temporarily store the physical addresses of all data sources obtained after decapsulating the data packets;

[0021] A calculation submodule, wherein the calculation submodule is used to filter the physical addresses in the address pool by calling the filtering layer through the filtering submodule, and mark abnormal physical addresses after filtering is completed;

[0022] An output module is used to output a judgment result according to the result of the calculation submodule.

[0023] Furthermore, the exception handling module locates the IoT device with abnormal transmission through strategy 2, and the strategy 2 is:

[0024] (1) Setting a filter layer, wherein the filter layer contains address information of all IoT devices in the current connection state of the gateway;

[0025] (2) Data decapsulation: decapsulate all data packets received within the current time period window T and obtain the physical addresses of all data sources;

[0026] (3) Filtering and positioning: all physical addresses obtained by decapsulation are filtered through the filtering layer, and abnormal physical addresses are marked. The marked abnormal physical addresses include physical addresses that are not enabled on the filtering layer and physical addresses that do not pass the filtering layer after decapsulation;

[0027] (4) abnormal marking is performed based on the judgment result. When it is judged that the abnormal transmission is caused by the suspension or damage of the IoT device, the IoT device corresponding to the physical address that is not enabled on the filter layer is marked as missing; when it is judged that the abnormal transmission is caused by the addition of an IoT device, the IoT device corresponding to the physical address that does not pass the filter layer after decapsulation is marked as new.

[0028] Furthermore, the exception handling module is also used to monitor the state change of abnormal transmission through strategy three, and the strategy three is jointly executed with strategy one and strategy two. The strategy three is:

[0029] identifying a physical address in the data marked missing as a target address;

[0030] When strategy 1 identifies an abnormal transmission and the abnormal transmission is a new transmission, strategy 2 marks the physical address that does not pass the filtering layer after decapsulation as a candidate address;

[0031] Compare the target address and the candidate address to see if they are consistent. If they are consistent, cancel the physical address marked as missing and the IoT device corresponding to the physical address;

[0032] If they are inconsistent, determine whether the newly transmitted data is of the same type as the data missing during the abnormal transmission. If they are of the same type, cancel the physical address marked as missing, and obtain the physical address of the newly transmitted data to update the original physical address; if they are not of the same type, keep the missing mark.

[0033] Further, the method for determining whether the newly transmitted data is of the same type as the data missing during abnormal transmission is as follows:

[0034] Obtaining a number of continuous data packets uploaded by the IoT device before the abnormal transmission, and further obtaining the unchanged data feature information in each data packet by decapsulation, and obtaining the timestamps in a number of continuous data and further obtaining the frequency information of data upload;

[0035] The data characteristic information that remains unchanged in the data packet and the frequency information of data upload are concatenated and arranged in a fixed combination to form a character string or a binary data block;

[0036] The concatenated data is hashed using a hash function to generate a fixed target hash value;

[0037] Obtaining a number of continuous data packets uploaded by the IoT device after the newly added transmission, and further obtaining the unchanged data feature information in each data packet by decapsulation, and obtaining the timestamps in a number of continuous data and further obtaining the frequency information of data upload;

[0038] The data characteristic information and the frequency information of data upload that remain unchanged in the data packet are spliced ​​and arranged in a fixed combination form to form a character string or a binary data block;

[0039] The concatenated data is hashed using a hash function to generate a fixed hash value to be verified;

[0040] Compare the target hash value with the hash value to be verified. If they are the same, the data are considered to be of the same type.

[0041] Furthermore, the calculation submodule is also used to compare the consistency of the target address and the candidate address by obtaining the timestamps in several continuous data and further obtaining the frequency information of data upload, splicing the data feature information and the data frequency information, performing hash operation to generate a hash value, and comparing the target hash value with the hash value to be verified.

[0042] Furthermore, a data transceiver module and a data processing module are also provided in the gateway.

[0043] Furthermore, the data processing module is used to upload the abnormal state of the Internet of Things device to the data collector, and the method is: inserting an abnormal state identification bit in the data packet, and representing different abnormal states by identifications with different codes.

[0044] The beneficial effects of the present invention are:

[0045] The present application sets an exception management module in the gateway, and through an overall analysis of the amount of data received during the operation of the gateway, combined with an analysis of the content and characteristics of specific data, determines whether the data transmission status of the IoT device connected to the gateway is abnormal. It can monitor the status changes of the IoT device from the data level and accurately feedback the back-end platform that collects data and processes and analyzes it, which is beneficial for the data collector to obtain more accurate data information. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Shown is the topological connection diagram of the data acquisition system. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments.

[0048] like Figure 1 As shown, it is a topological connection diagram of the data acquisition system in this application. The data acquisition system includes a data collector, a gateway, and an Internet of Things device. The data collector can communicate data with multiple gateways, and the same gateway can also communicate data with multiple Internet of Things devices to achieve systematic management of data. The Internet of Things device is used to collect data and transmit the collected data to the gateway through a specified transmission protocol. The gateway processes the data according to the data management strategy and sends it to the data collector.

[0049] The data communication mentioned above refers to the process of exchanging information between two devices through some form of transmission media. The achievable data communication includes wired communication and wireless communication. Wired communication includes achievable Ethernet transmission, optical fiber transmission, coaxial cable transmission, etc. Wireless communication further includes short-range wireless communication and long-range wireless communication. Short-range wireless communication circuits include Bluetooth, Wi-Fi, Zigbee, NFC, etc., and long-range wireless communication includes 2G, 3G, 4G / LTE and 5G and other mobile communication technologies.

[0050] IoT devices include various sensors to collect or receive IoT data from terminal devices. Sensors include but are not limited to motion detection, radio frequency identification (RFID), flow detection, location detection, weight detection, signal detection and other sensors. Sensors also include infrared sensors, global positioning systems, laser scanners, gas sensors and other information sensing devices.

[0051] The gateway is provided with a data transceiver module, an exception management module, an exception processing module and a data processing module.

[0052] The data transceiver module is a common module used to realize uplink and downlink data transmission. The transceiver module is selected according to the specific mode of data communication, and can be set up and used alone or in combination. For example, an optical fiber communication interface can be set for wired communication, or a Wi-Fi module can be set to transmit and receive data within the communication range.

[0053] The data processing module is used to perform basic data encapsulation, decapsulation, load identification and other operations.

[0054] The setting of the exception management module is used to identify abnormal situations of data transmission during the working process, and at the same time provide feedback processing to the data collector. The abnormal situations of data transmission described in this application are mainly caused by the maintenance suspension, damage replacement and addition of IoT devices. For example, when an IoT device is suspended for maintenance, the gateway will also correspondingly lack the data information uploaded by the IoT device during the data sending and receiving management process.

[0055] The core function of the exception management module is to identify abnormal data transmission, which mainly involves the following strategy 1:

[0056] (1) Setting a time period window T, which is relatively short and is usually in seconds;

[0057] (2) Continuously counting the amount of data received based on the time period window T to obtain the total amount of data received S1 within the time period window T;

[0058] (3) Retrieve the total amount of data received S2 counted in the previous time period window T, and compare the total amount of data received S1 in the current time period window T with the total amount of data received S2 in the previous time period window T, and output a judgment result based on the comparison result:

[0059] If S1 is greater than S2, the abnormal transmission is judged to be a new IoT device;

[0060] If S1 is less than S2, the abnormal transmission is judged to be due to the suspension of use or damage of the IoT device;

[0061] If S1 is equal to S2, it is determined that there is no abnormal transmission.

[0062] Since the IoT devices connected to the gateway are relatively stable and their working states are single, the frequency of data uploaded by the IoT devices and the size of data packets are also relatively stable. Once an IoT device is disabled or added, the maximum instantaneous data reception amount of the gateway will increase or decrease. However, since the IoT devices connected to the gateway are not exactly the same, the data states uploaded by different IoT devices are also different. Therefore, the instantaneous data reception amount does not change significantly at any time. Therefore, setting a time period window T and counting the total statistical data reception amount S1 within the time period window T can not only amplify the change in the data reception amount, so as to better identify and judge abnormal transmission, but also the time period window T can greatly reduce the frequency of data statistics, calculation, and processing, thereby avoiding the increase in gateway workload caused by the high frequency operation of the abnormal management module.

[0063] Accordingly, in order to realize the purpose of identifying abnormal data transmission, the abnormal management module should be provided with corresponding functional submodules, including a time period window submodule, a statistics submodule, a storage submodule and a judgment submodule.

[0064] The time period window submodule can set the time period window T according to different application scenarios and the status of data uploaded by the IoT device to which it is connected, so as to better manage the identification of abnormal transmission.

[0065] The statistics submodule performs periodic statistics of data reception by calling the time period window T;

[0066] The storage submodule is used to store the statistical data received in different time period windows T;

[0067] The judgment submodule obtains the total amount of data received in the current time period window T and the total amount of data received in the previous time period window T, performs a simple numerical comparison, and outputs a judgment result based on the comparison result.

[0068] The exception handling module is activated when the exception management module identifies abnormal data transmission. Its purpose is to locate the IoT device where abnormal transmission occurs. This process mainly involves the following strategy 2:

[0069] (1) Setting a filter layer, which contains the address information of all IoT devices in the gateway's current connection state, including physical addresses and network addresses;

[0070] (2) Data decapsulation: decapsulate all data packets received within the current time period window T and obtain the physical addresses of all data sources;

[0071] (3) Filtering and positioning: all physical addresses obtained by decapsulation are filtered through the filtering layer, and abnormal physical addresses are marked. The marked abnormal physical addresses include physical addresses that are not enabled on the filtering layer and physical addresses that do not pass the filtering layer after decapsulation;

[0072] (4) abnormal marking is performed based on the judgment result. When it is judged that the abnormal transmission is caused by the suspension of use or damage of the IoT device, the IoT device corresponding to the physical address that is not enabled on the filter layer is marked as missing, and the corresponding device is suspended or damaged; when it is judged that the abnormal transmission is caused by the addition of an IoT device, the IoT device corresponding to the physical address that does not pass the filter layer after decapsulation is marked as added.

[0073] In addition, the exception handling module also sets a strategy three in addition to the above strategies. This strategy three is enabled when a missing mark is formed. The role of this strategy three is to monitor the state changes of abnormal transmission to facilitate more comprehensive data management. This strategy three is jointly executed with strategy one and strategy two. The content is:

[0074] identifying a physical address in the data marked missing as a target address;

[0075] When strategy 1 identifies an abnormal transmission and the abnormal transmission is a new transmission, strategy 2 marks the physical address that does not pass the filtering layer after decapsulation as a candidate address;

[0076] Compare the target address and the candidate address to see if they are consistent. If they are consistent, cancel the physical address marked as missing and the IoT device corresponding to the physical address. This indicates that the abnormal transmission is caused by the suspension of the IoT device due to scheduled maintenance. The device will be reactivated after the maintenance is completed.

[0077] If they are inconsistent, determine whether the newly transmitted data is of the same type as the data that was missing during the abnormal transmission. If they are of the same type, cancel the physical address marked as missing, and obtain the physical address of the newly transmitted data to update the original physical address. This indicates that the abnormal transmission occurred because the IoT device may be damaged and suspended, and it will be reactivated after replacing a new IoT device. If they are not of the same type, the missing mark is still maintained, indicating that the IoT device with abnormal transmission has not been restored to use through other means.

[0078] Regarding the above-mentioned determination of whether the newly transmitted data is of the same type as the data missing during abnormal transmission, the method provided by this application is:

[0079] Obtaining a number of continuous data packets uploaded by the IoT device before the abnormal transmission, and further obtaining unchanged data feature information in each data packet by decapsulation, including but not limited to data type, encapsulation protocol, and payload length, and obtaining timestamps in a number of continuous data and further obtaining frequency information of data upload;

[0080] The data characteristic information and the frequency information of data upload that remain unchanged in the data packet are concatenated and arranged in a fixed combination to form a character string or a binary data block, such as concatenating the frequency of data upload, data type, encapsulation protocol, and effective load length;

[0081] The concatenated data is hashed using a hash function to generate a fixed target hash value;

[0082] Obtaining a number of continuous data packets uploaded by the IoT device after the newly added transmission, and further obtaining the unchanged data feature information in each data packet by decapsulation, including but not limited to the data type, encapsulation protocol, and effective load length, and obtaining the timestamps in a number of continuous data and further obtaining the frequency information of data upload;

[0083] The data characteristic information and the frequency information of data upload that remain unchanged in the data packet are spliced ​​and arranged in a fixed combination form to form a string or a binary data block, and the combination form is the same as the above, for example, splicing with the frequency of data upload, data type, encapsulation protocol, and effective load length;

[0084] The concatenated data is hashed using a hash function to generate a fixed hash value to be verified;

[0085] Compare the target hash value with the hash value to be verified. If they are the same, the data are considered to be of the same type.

[0086] Accordingly, in order to locate the IoT device with abnormal transmission, the exception handling module should be provided with corresponding functional submodules, including:

[0087] A filtering submodule, which can call the routing table and ARP cache table in the gateway to fill the physical addresses stored therein into the filtering layer;

[0088] The filter layer generated by the filter submodule in the present application changes according to the contents of the routing table and the ARP cache table, and the address information in the routing table and the ARP cache table will also change according to the application status of the IoT device. When the entire data acquisition system remains stable, the routing table and the ARP cache table also remain stable.

[0089] An address pool, which is used to temporarily store the physical addresses of all data sources obtained after decapsulating the data packets;

[0090] A calculation submodule, which is used to filter the physical addresses in the address pool by calling the filtering layer through the filtering submodule, and mark abnormal physical addresses after filtering; the calculation submodule is also used to compare the consistency of the target address and the candidate address; and to obtain the timestamps in a number of continuous data and further obtain the frequency information of data upload; and to generate a hash value by performing a hash operation after splicing the data feature information and the data frequency information; and to compare the target hash value with the hash value to be verified;

[0091] The output module is used to output the judgment results according to different calculation or comparison results in different strategies.

[0092] The physical address described in this application refers to the address of the data link layer, that is, the MAC address.

[0093] On the other hand, the data processing module in the present application is not only used to implement basic operations such as encapsulation, decapsulation, and load identification, but also used to upload the abnormal status of the Internet of Things device to the data collector. When verifying the data packet and re-encapsulating it, the data processing module will insert an abnormal status identification bit in the data packet, and use different encoded identifications to represent the four states of addition, deletion, recovery and replacement. In addition, a default data packet format will be defined between the gateway and the data collector. When a missing state occurs, the data processing module will enable the default data packet and upload an empty packet data to the data collector through the format of the default data packet to inform the corresponding transmission of the missing state of the Internet of Things device. The three states of addition, recovery and replacement can directly transmit the original data packet. It is only necessary to insert the abnormal status identification bit to identify the three states of addition, recovery and replacement. When the abnormal status identification bit is the default value, it indicates that the system data collection is normal, and there is no addition, deletion, recovery or replacement state.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them.

Claims

1. A data acquisition system based on an Internet of Things device, comprising an Internet of Things device arranged at the front end, a gateway connected to the Internet of Things device for communication, and a data collector connected to the gateway for communication, characterized in that: The gateway is provided with an abnormality management module, which is used to identify abnormal situations of data transmission during operation, and the abnormality management module includes: A time period window submodule, wherein the time period window submodule is used to set a time period window T; A statistical submodule, wherein the statistical submodule performs statistics on the amount of periodic data received by calling a time period window T; A storage submodule, the storage submodule is used to store the amount of data received statistically in different time period windows T; The judgment submodule obtains the total amount of data received in the current time period window T and the total amount of data received in the previous time period window T, compares them, and outputs a judgment result according to the comparison result.

2. The data acquisition system based on Internet of Things devices according to claim 1 is characterized in that: The abnormal management module identifies abnormal data transmission through strategy 1, and the strategy 1 is: (1) Set a time period window T; (2) Continuously counting the amount of data received based on the time period window T to obtain the total amount of data received S1 within the time period window T; (3) Retrieve the total amount of data received S2 counted in the previous time period window T, and compare the total amount of data received S1 in the current time period window T with the total amount of data received S2 in the previous time period window T, and output a judgment result based on the comparison result: If S1 is greater than S2, the abnormal transmission is judged to be a new IoT device; If S1 is less than S2, the abnormal transmission is judged to be due to the suspension of use or damage of the IoT device; If S1 is equal to S2, it is determined that there is no abnormal transmission.

3. The data acquisition system based on Internet of Things devices according to claim 1 is characterized in that: The gateway is also provided with an exception handling module, which is enabled when the exception management module identifies abnormal data transmission, and is used to locate the IoT device where abnormal transmission occurs. The exception handling module includes: A filtering submodule, wherein the filtering submodule can call the routing table and the ARP cache table in the gateway, thereby filling the physical addresses stored therein into the filtering layer; An address pool, the address pool is used to temporarily store the physical addresses of all data sources obtained after decapsulating the data packets; A calculation submodule, wherein the calculation submodule is used to filter the physical addresses in the address pool by calling the filtering layer through the filtering submodule, and mark abnormal physical addresses after filtering is completed; An output module is used to output a judgment result according to the result of the calculation submodule.

4. The data acquisition system based on Internet of Things devices according to claim 3 is characterized in that: The exception handling module locates the IoT device with abnormal transmission through strategy 2, and the strategy 2 is: (1) Setting a filter layer, wherein the filter layer contains address information of all IoT devices in the current connection state of the gateway; (2) Data decapsulation: decapsulate all data packets received within the current time period window T and obtain the physical addresses of all data sources; (3) Filtering and positioning: all physical addresses obtained by decapsulation are filtered through the filtering layer, and abnormal physical addresses are marked. The marked abnormal physical addresses include physical addresses that are not enabled on the filtering layer and physical addresses that do not pass the filtering layer after decapsulation; (4) marking anomalies in combination with the judgment result. When it is judged that the abnormal transmission is caused by the suspension or damage of the IoT device, the IoT device corresponding to the physical address that is not enabled on the filter layer is marked as missing; When it is determined that the abnormal transmission is a newly added IoT device, the IoT device corresponding to the physical address that fails to pass the filtering layer after decapsulation is marked as newly added.

5. The data acquisition system based on Internet of Things devices according to claim 4 is characterized in that: The exception handling module is also used to monitor the state change of abnormal transmission through strategy three, and the strategy three is jointly executed with strategy one and strategy two. The strategy three is: identifying a physical address in the data marked missing as a target address; When strategy 1 identifies an abnormal transmission and the abnormal transmission is a new transmission, strategy 2 marks the physical address that does not pass the filtering layer after decapsulation as a candidate address; Compare the target address and the candidate address to see if they are consistent. If they are consistent, cancel the physical address marked as missing and the IoT device corresponding to the physical address; If they are inconsistent, determine whether the newly transmitted data is of the same type as the data missing during the abnormal transmission. If they are of the same type, cancel the physical address marked as missing, and obtain the physical address of the newly transmitted data to update the original physical address; if they are of different types, keep the missing mark.

6. The data acquisition system based on Internet of Things devices according to claim 5 is characterized in that: The method for determining whether the newly transmitted data is of the same type as the data missing during abnormal transmission is as follows: Obtaining a number of continuous data packets uploaded by the IoT device before the abnormal transmission, and further obtaining the unchanged data feature information in each data packet by decapsulation, and obtaining the timestamps in a number of continuous data and further obtaining the frequency information of data upload; The data characteristic information that remains unchanged in the data packet and the frequency information of data upload are concatenated and arranged in a fixed combination to form a character string or a binary data block; The concatenated data is hashed using a hash function to generate a fixed target hash value; Obtaining a number of continuous data packets uploaded by the IoT device after the newly added transmission, and further obtaining the unchanged data feature information in each data packet by decapsulation, and obtaining the timestamps in a number of continuous data and further obtaining the frequency information of data upload; The data characteristic information and the frequency information of data upload that remain unchanged in the data packet are spliced ​​and arranged in a fixed combination form to form a character string or a binary data block; The concatenated data is hashed using a hash function to generate a fixed hash value to be verified; Compare the target hash value with the hash value to be verified. If they are the same, the data are considered to be of the same type.

7. The data acquisition system based on Internet of Things devices according to claim 6 is characterized in that: The calculation submodule is also used to compare the consistency of the target address and the candidate address by obtaining the timestamps in several continuous data and further obtaining the frequency information of data upload, splicing the data feature information and the data frequency information, performing hash operation to generate a hash value, and comparing the target hash value with the hash value to be verified.

8. The data acquisition system based on Internet of Things devices according to claim 1 is characterized in that: The gateway is also provided with a data transceiver module and a data processing module.

9. The data acquisition system based on Internet of Things devices according to claim 8, characterized in that: The data processing module is used to upload the abnormal state of the Internet of Things device to the data collector, and the method is: inserting an abnormal state identification bit in the data packet, and representing different abnormal states by identifications with different codes.

Images