A method for efficiently caching historical data in offline state of IoT data collection gateway

Abstract

The present invention discloses a method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway, including a deployment relationship of the Internet of Things data collection gateway, which includes field instruments, the Internet of Things data collection gateway, and a cloud server. The data of the field instruments are collected by the Internet of Things data collection gateway and temporarily stored inside the Internet of Things data collection gateway, and the data is uploaded to the cloud server through communication methods such as network transmission; users do not need to care about the size of memory and disk space, the cache engine storage efficiency is not high, resulting in a small amount of cached data, and the disk will be full in a long-term offline state.

Description

Technical Field

[0001] The invention belongs to the technical field of Internet of Things, and in particular relates to a method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway. Background Art

[0002] The IoT data acquisition gateway, also called the industrial IoT smart gateway or wireless data acquisition gateway, is a wireless sensor network product. It is highly integrated and can collect data, perform protocol conversion, forward data, and transmit wireless communications to the connected sensor data. It supports multiple communication protocols and methods, including 4G, Wi-Fi, Ethernet, and other communication methods.

[0003] IoT data collection gateways have the following characteristics: limited processing power, small amounts of memory and disk space, and fragile wireless conditions.

[0004] In the process of data forwarding, the IoT data collection gateway often encounters poor communication status. In this case, the data cannot be forwarded to the cloud, and the data integrity that the cloud application relies on cannot be met.

[0005] IoT data collection gateways usually have limited memory and disk space. Due to the low storage efficiency of the cache engine, the amount of cached data is very small. At the same time, if the device is offline for a long time, the disk will be full. Summary of the invention

[0006] The purpose of the present invention is to provide a method for caching historical data when the IoT data acquisition gateway is offline, so as to solve the problem that the IoT data acquisition gateway usually has limited memory and disk space, the storage efficiency of the cache engine is not high, the amount of cached data is very small, and the disk will be full when offline for a long time.

[0007] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0008] The object of the present invention is to provide an efficient method for caching historical data in an offline state of an Internet of Things data collection gateway, including an Internet of Things data collection gateway deployment relationship, a cache file creation and cache file deletion stage, a cache file composition stage, a cache data encoding and compression stage, and a cache record reading stage;

[0009] The deployment relationship of the Internet of Things data collection gateway includes field instruments, Internet of Things data collection gateway, and cloud servers. The data of the field instruments are collected by the Internet of Things data collection gateway and temporarily stored inside the Internet of Things data collection gateway. The data is uploaded to the cloud server through communication methods such as network transmission;

[0010] When the data of the field instrument needs to be cached, if the cache file is not full, the writing operation can continue, and the next data will be written additionally to the cache file used this time. If the cache file is full, the current cache file needs to be closed and compressed and archived. At the same time, it is determined whether the capacity of the current overall cache file has reached the upper limit value. If the upper limit value has not been reached, a new cache file is created to continue writing data. If the upper limit value is reached, the earliest archived cache file in the existing file is deleted, and a new cache file is created to continue writing data.

[0011] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data acquisition gateway is provided. The Internet of Things data acquisition gateway will continue to temporarily store the data of the field instruments when the network is unstable, and will upload the cached data to a cloud server after the network is restored.

[0012] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data acquisition gateway is provided. When the Internet of Things data acquisition gateway resumes uploading, it will search the folder to determine whether there is a cache file. If there is no cache file, the data upload will be resumed. If there is a cache file, the cache file with the smallest number in the cache file will be searched, and it will be determined whether it is a zip file. If it is a zip file, it needs to be decompressed and opened. If not, the cache file is directly opened, and the file header information of the cache file is further verified, and the file index information area is searched to find the first index information item that has not been marked for deletion, and the cache data is read according to the index information, offset and number of bytes, and the cache data is verified. After the cache data is read, the deletion identification item of the index information of the cache file will be updated accordingly, and the cache data will be decompressed, and then decoded through variable-length decoding to obtain each attribute value of the cache data, and returned to the user at the same time, and uploaded to the cloud server through network transmission.

[0013] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data acquisition gateway is provided, wherein the cache file is provided with file header information, a record index information area and a record data information area. When a new cache file is created, the file header information is first written, and then the basic information of the index block of the cache file is created according to the set number of records that the file can store.

[0014] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data acquisition gateway is provided, wherein the file header information of the cache file contains three elements, a fixed value, a file capacity, and the number of used records. The fixed value serves to determine whether it is a cache file. The file capacity can be a preset value. When the number of used records reaches the file capacity, it indicates that the cache file has been filled.

[0015] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway is provided, wherein the record index information area has multiple index information, and the number of index information can be seen based on a preset value.

[0016] Furthermore, a method for efficiently caching historical data in an offline state of an IoT data collection gateway is provided, wherein the index information is provided with an offset position of the cache record in the file, the number of bytes of the cache record, a cyclic redundancy check (CRC) value of the cache record, and whether it is marked as a deleted state. The deleted state means that after the IoT data collection gateway resumes the networking state, the cache record will be read from the cache file and the read record will be uploaded to the cloud server. At this time, the record is in a deleted state until all the cache files are read, and the cache file will be deleted.

[0017] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway is provided, wherein the record data information area has multiple cache record data, and the number of cache record data is the same as the number of index information. Before writing the cache record data, the written cache record data will be variable-length encoded, and then the encoded bit array will be losslessly compressed using the LZ77 algorithm and Huffman coding. The compressed lossless compressed data will be written into the cache record area of ​​the cache file, and the file header information and index information of the current cache file will be updated at the same time.

[0018] Furthermore, a method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway is provided. The cache record data has multi-attribute information. The present invention performs infinitive long encoding according to different types of cache record data to reduce space occupancy, and each data corresponds to a different numerical value. Finally, the encoded cache record data is compressed as cache file data.

[0019] The present invention has the following beneficial effects:

[0020] For the data of all the collection points that need to be cached, variable-length encoding is performed according to the data type to reduce storage space. The encoded data is compressed using a lossless data compression algorithm based on the LZ77 algorithm and Huffman coding, and a cache file containing index information blocks and data blocks is established. The compressed data above is appended to the file database in sequence each time, and the index information of the file is updated at the same time. When the index information of the cache file is full, the cache file is compressed by zip archive. If all the data in the cache file is marked as deleted or the disk space is limited, the earliest historical cache file will be deleted.

[0021] The historical data caching method of the present invention performs variable-length encoding and multi-level compression on the historical data, thereby reducing the storage overhead of the disk and increasing the storage capacity. A cyclic redundancy check is added to the written content to ensure the reliability of the data. In addition, when the file is full or the disk space reaches the upper limit, the expired cache file is automatically deleted, avoiding manual data deletion operations and reducing the maintenance cost of the data acquisition gateway.

[0022] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 : The deployment location of the Internet of Things data collection gateway of the present invention

[0025] Figure 2 : The cache file creation and deletion logic diagram of the present invention

[0026] Figure 3 : Cache file composition structure diagram of the present invention

[0027] Figure 4 : The encoding and compression stage diagram of the cache data of the present invention

[0028] Figure 5 : Cache record reading logic diagram of the present invention DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inside", "all around" and the like indicating orientation or positional relationship are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0031] like Figure 1-5As shown: A method for efficiently caching historical data in an offline state of an IoT data collection gateway.

[0032] It includes the IoT data collection gateway deployment relationship, cache file creation and cache file deletion phase, cache file composition phase, cache data encoding and compression phase, and cache record reading phase;

[0033] The deployment relationship of the Internet of Things data collection gateway includes field instruments, Internet of Things data collection gateway, and cloud servers. The data of the field instruments are collected by the Internet of Things data collection gateway and temporarily stored inside the Internet of Things data collection gateway. The data is uploaded to the cloud server through communication methods such as network transmission;

[0034] When the data of the field instrument needs to be cached, if the cache file is not full, the writing operation can continue, and the next data will be written additionally to the cache file used this time. If the cache file is full, the current cache file needs to be closed and compressed and archived. At the same time, it is determined whether the capacity of the current overall cache file has reached the upper limit value. If the upper limit value has not been reached, a new cache file is created to continue writing data. If the upper limit value is reached, the earliest archived cache file in the existing file is deleted, and a new cache file is created to continue writing data.

[0035] like Figure 1-5 As shown: the IoT data acquisition gateway will continue to temporarily store the data of the field instrument when the network is unstable, and will upload the cached data to the cloud server after the network is restored.

[0036] like Figure 1-5 As shown: when the IoT data acquisition gateway resumes uploading, it will search the folder to determine whether there is a cache file. If there is no cache file, the data upload will be resumed. If there is a cache file, the cache file with the smallest number in the cache file will be searched, and it will be determined whether it is a zip file. If it is a zip file, it needs to be decompressed and opened. If not, the cache file will be directly opened, and the file header information of the cache file will be verified, and the file index information area will be searched to find the first index information item that has not been marked for deletion, and the cache data will be read according to the index information, offset and number of bytes, and the cache data will be verified. After the cache data is read, the deletion identification item of the index information of the cache file will be updated, and the cache data will be decompressed, and then decoded through indefinite length decoding to obtain each attribute value of the cache data, and returned to the user at the same time, and uploaded to the cloud server through network transmission.

[0037] like Figure 1-5As shown: the cache file is provided with file header information, record index information area and record data information area. When a new cache file is created, the file header information is first written, and then the basic information of the index block of the cache file is created according to the number of records that the set file can store.

[0038] like Figure 1-5 As shown: the file header information of the cache file contains three elements, a fixed value, a file capacity and the number of used records. The fixed value is used to determine whether it is a cache file. The file capacity can be a preset value. When the number of used records reaches the file capacity, it means that the cache file has been filled.

[0039] like Figure 1-5 As shown: the record index information area has multiple index information, and the number of index information can be seen according to the preset value.

[0040] like Figure 1-5 As shown: the index information is provided with the offset position of the cache record in the file, the number of bytes of the cache record, the cyclic redundancy check (CRC) value of the cache record and whether it is marked as a deleted state.

[0041] like Figure 1-5 As shown: the record data information area contains multiple cache record data, and the number of cache record data is the same as the number of index information. Before writing the cache record data, the written cache record data will be variable-length encoded, and then the encoded bit array will be subjected to lossless data compression using the LZ77 algorithm and Huffman coding. The compressed lossless compressed data will be written into the cache record area of ​​the cache file, and the file header information and index information of the current cache file will be updated at the same time.

[0042] like Figure 1-5 As shown: the cache record data has multiple attribute information. The present invention will perform indefinite long encoding according to different types of cache record data to reduce space occupancy, and each data corresponds to a different value. Finally, the encoded cache record data is compressed as cache file data.

[0043] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0044] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway, characterized in that: It includes the IoT data collection gateway deployment relationship, cache file creation and cache file deletion phase, cache file composition phase, cache data encoding and compression phase, and cache record reading phase; The deployment relationship of the Internet of Things data collection gateway includes field instruments, Internet of Things data collection gateway, and cloud servers. The data of the field instruments are collected by the Internet of Things data collection gateway and temporarily stored inside the Internet of Things data collection gateway, and the data is uploaded to the cloud server through network transmission communication; When the data of the field instrument needs to be cached, if the cache file is not full, the writing operation will continue, and the next data will be written into the cache file used this time. If the cache file is full, the current cache file needs to be closed and compressed and archived. At the same time, it is determined whether the capacity of the current overall cache file has reached the upper limit value. If the upper limit value has not been reached, a new cache file will continue to be created to continue writing data. If the upper limit value is reached, the earliest archived cache file in the existing file will be deleted, and a new cache file will be created to continue writing data; The IoT data collection gateway will temporarily store the data of the field instrument when the network is unstable, and will actively upload the cached data to the cloud server after the network is restored; When the IoT data acquisition gateway resumes uploading, it will search the cache directory to determine whether there is a cache file. If there is no cache file, the cache data upload will be terminated. If there is a cache file, the cache file with the smallest number in the cache file will be searched, and it will be determined whether it is a zip file. If it is a zip file, it needs to be decompressed and opened. If not, the cache file will be directly opened, and the file header information of the cache file will be verified, and the file index information area will be searched to find the first index information item that has not been marked for deletion, and the cache data will be read according to the index information, offset and number of bytes, and the cache data will be verified. After the cache data is read, the deletion identification item of the index information of the cache file will be updated, and the cache data will be decompressed, and then decoded by indefinite length decoding to obtain each attribute value of the cache data, and returned to the user at the same time, and uploaded to the cloud server through network transmission; The cache file is provided with file header information, record index information area and record data information area. When a new cache file is created, the file header information is first written, and then the index block basic information of the cache file is created according to the number of records stored in the set file; The file header information of the cache file includes three elements: a fixed value, a file capacity, and the number of used records. The fixed value is used to determine whether it is a cache file. The file capacity is a preset value. When the number of used records reaches the file capacity, it means that the cache file has been filled. The record index information area has a plurality of index information, and the number of index information is determined according to a preset value; The index information is provided with the offset position of the cache record in the file, the number of bytes of the cache record, the cyclic redundancy check (CRC) value of the cache record and whether it is marked as a deleted state; The record data information area contains multiple cache record data, and the number of cache record data is the same as the number of index information. Before writing the cache record data, the written cache record data will be variable-length encoded, and then the encoded bit array will be losslessly compressed using the LZ77 algorithm and Huffman coding. The compressed lossless compressed data will be written into the cache record area of ​​the cache file, and the file header information and index information of the current cache file will be updated at the same time.

2. According to claim 1, a method for efficiently caching historical data in an offline state of an Internet of Things data collection gateway is characterized by: The cache record data has multiple attribute information, and will be encoded in an infinitive long encoding according to the different data types of the attribute values ​​of each cache record to reduce space occupancy, and each data record sub-item has a corresponding different numerical type. Finally, the encoded cache record data is compressed as the data of the cache file.

Images