High-frequency industrial time series data storage system based on double caching

Abstract

The present invention provides a high-frequency industrial time series data storage system based on double caching. A double caching strategy is used to isolate data acquisition and data storage, the structure is simple, and the modules are clear, ensuring that the data processing process is not blocked and other associated modules are not affected, improving the overall stability of the system, ensuring the reliability and timeliness of data storage, and achieving the function of storing time series data in massive, high-concurrency, high-frequency, and other scenarios. The present invention has higher performance and better expandability. By means of hierarchical design and reasonable functional module division, the requirements of high throughput, low delay and high expandability are met, thereby better supporting the application of industrial time series data. According to the present invention, a message layer and a caching layer are introduced, achieving real-time data processing and caching, improving the data processing efficiency, and reducing the data processing delay.

Description

A high-frequency industrial time series data storage system based on double caching Technical Field

[0001] The present invention belongs to the technical field of industrial big data platforms, and in particular relates to a high-frequency industrial time series data storage system based on double caching. Background Art

[0002] Industrial time series data refers to the large amounts of time series data generated during industrial production. This data primarily includes equipment operating status data, sensor data, and production process data. With the increasing degree of industrial automation, the collection, storage, and analysis of industrial time series data are becoming increasingly important. Generally speaking, in scenarios where the collection frequency is low and the data volume is small, direct collection and storage is feasible.

[0003] With the continuous development of industrial production, the speed and scale of industrial time series data generation are growing exponentially, placing higher demands on data storage and processing. However, existing industrial time series data storage solutions have some shortcomings. For example, they often fail to meet the requirements of high throughput, low latency, and high scalability, which limits the application scope of industrial time series data.

[0004] The existing time series data storage method for industrial Internet of Things cleans the time series data by obtaining the time series data to be stored; assigns the category topic of the time series data according to the source of the time series data, and writes the cleaned time series data to the partition corresponding to the topic in the message distribution component Kafka. During the writing process, the collection timestamp corresponding to each time series data and the offset information of each time series data in the partition are cached; the offset information under each partition is sorted and merged according to the time indicated by the collection timestamp, and the association relationship between the topic, collection timestamp and offset information is determined; the association relationship between the topic, collection timestamp and offset information is persisted to the index library.

[0005] An existing method for data collection using the Modbus industrial bus field protocol employs Modbus as the data collection protocol, MQTT as the configuration file distribution and data storage solution, InfluxDB as the time series database for storage, Java as the code, and JSON as the format for configuration files and API interaction. This method implements data collection, data storage, and data visualization, meeting the needs of enterprises for data collection and intelligent production in the Industrial Internet of Things (IIoT) environment, and enabling data collection from edge devices in the cloud.

[0006] The above-mentioned public technologies have improved the storage performance of industrial time series data to a certain extent, but have not solved the storage problems of time series data in scenarios such as massive, high concurrency, and high frequency.

[0007] Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a high-frequency industrial time series data storage system based on double caching, which is used to store time series data in scenarios such as massive, high concurrency, and high frequency.

[0009] The technical solution adopted by the present invention to solve the above technical problems is: a high-frequency industrial time series data storage system based on double caching, including an acquisition layer, a message layer, a cache layer, a data processing layer and a storage layer connected in sequence; the acquisition layer is a platform that supports multiple data acquisition drivers, used to collect real-time data from equipment and sensors at industrial production sites and support asynchronous push to the message layer via MQTT, providing a single source of industrial automation data to applications; the message layer adopts a distributed cluster architecture, including an MQTT broker, supporting protocols including MQTT and MQTTs, for receiving messages from the acquisition layer and sending them to the data processing layer; the message layer also includes a Web Socket broker, supporting protocols including WS and WSS, for receiving messages from the data processing layer and sending them to the HMI screen of the external system for use; the cache layer includes a first cache module and a second cache module for providing double caching for the data processing layer; the first cache module is a topic and payload cache, used to prevent the data processing layer from consuming message layer data in a timely manner and causing congestion; the second cache module is a business data cache, used to prevent the data processing layer from saving data in a timely manner and causing congestion; the storage layer includes a time series database.

[0010] According to the above scheme, the industrial communication protocols used by the acquisition layer include Modbus, OPC-UA or PLC-specific protocols; the equipment and sensors at the industrial production site are external systems including PLCs, RTUs, related instruments or third-party systems; the data pushed by the acquisition layer to the message layer is encoded in JSON format.

[0011] According to the above solution, the message layer uses the open source EMQX, which is an elastically scalable cloud-native distributed IoT MQTT / WS message server. It is used to connect multiple IoT devices, move and process message and event stream data in real time, and build mission-critical IoT platforms and applications.

[0012] According to the above scheme, the cache uses memory cache, Redis cache or Memcached cache; the cache adopts a first-in-first-out FIFO cleaning strategy to clean up data according to the time when the data enters the cache; when the cache space is insufficient, the earliest data entering the cache is automatically deleted.

[0013] According to the above scheme, the data processing layer includes a subscription module, a first cache processing module and a second cache processing module; the subscription module is triggered to run, and the first cache processing module and the second cache processing module are both run periodically; the subscription module is used to subscribe to the MQTT data of the message layer; the first cache processing module is used to obtain and process data from the first cache module; the second cache processing module is used to obtain business data from the second cache module and store it in batches into the time series database of the storage layer.

[0014] Furthermore, the specific steps for the subscription module to subscribe to the MQTT data at the message layer are as follows:

[0015] S411: Filter data when some received topics need to be removed due to wildcard subscription and when no messages need to be received according to business logic;

[0016] S412: Asynchronously storing the topic and payload data into the first cache module for valid data.

[0017] Furthermore, the specific steps of the first cache processing module obtaining and processing data from the first cache module are:

[0018] S421: Data parsing, converting payload data into readable data;

[0019] S422: Data tracking, tracking and converting the original data into business data;

[0020] S423: asynchronously push the business data to the Web Socket broker of the message layer;

[0021] S424: Storing the business data in the second cache module.

[0022] Furthermore, the specific steps of the second cache processing module obtaining business data from the second cache module and storing the data in batches into the time series database of the storage layer are as follows:

[0023] S431: Acquire business data of the second cache module;

[0024] S432: Clear the business data acquired by the second cache module;

[0025] S433: Batch storage to the time series database of the storage layer;

[0026] S434: If the storage is successful, the single process ends; if the storage fails, the business data that failed to be stored is stored again in the second cache module.

[0027] According to the above solution, the storage layer is used to persistently store business data processed by the data processing layer; the storage layer uses the InfluxDB time series database, distributed file system or columnar database, including HDFS or Cassandra; the storage layer adopts a distributed cluster architecture to ensure storage performance and data security.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention discloses a high-frequency industrial time series data storage system based on double caching. By adopting a double-caching strategy, it isolates data acquisition and storage, has a simple structure, and clear modules. This ensures that the data processing process is not blocked and does not affect other related modules, thereby improving the overall stability of the system, ensuring the reliability and timeliness of data storage, and realizing the function of storing time series data in scenarios such as massive, high concurrency, and high frequency.

[0030] 2. Compared with existing industrial time series data storage solutions, this invention has higher performance and better scalability. Through layered design and reasonable functional module division, it meets the requirements of high throughput, low latency, and high scalability, thereby better supporting the application of industrial time series data.

[0031] 3. This invention adopts a layered architecture that separates data acquisition, processing, and storage, reducing system complexity and improving system maintainability and scalability. The acquisition layer enables high-throughput, low-latency data acquisition; the messaging layer enables reliable data transmission and inter-system interaction; the cache layer accelerates data access and processing; the data processing layer parses, tracks, and saves data; and the storage layer enables highly scalable storage, ensuring data security and reliability.

[0032] 4. This invention employs a dual-caching strategy. The first caching strategy addresses the issue of delayed data consumption after subscription, preventing data congestion. The second caching strategy addresses data congestion caused by delayed data storage at the data processing layer. This strategy also ensures orderly data storage, preventing thread conflicts and supporting a retry mechanism for data storage failures. Overall, the introduction of the message layer and cache layer enables real-time data processing and caching, improving data processing efficiency and reducing data processing latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a principle block diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Referring to Figure 1 , an embodiment of the present invention includes a collection layer, a message layer, a cache layer, a data processing layer, and a storage layer;

[0036] 1. Collection layer: responsible for collecting real-time data from equipment and sensors at industrial production sites.

[0037] The acquisition layer can use existing industrial communication protocols, such as Modbus, OPC-UA, and mainstream PLC-specific protocols, to collect data from various external systems, including but not limited to PLCs, RTUs, related instruments, and third-party systems.

[0038] In addition, the collection layer needs to have the function of pushing the collected data to the message layer.

[0039] Preferably, the pushed data is encoded in JSON format.

[0040] In a specific embodiment, the collection layer generally uses Kepware's KEPServerEX software, which is an industry-leading connection platform that contains a large number of data collection drivers and is used to provide a single source of industrial automation data to all applications. The collection results can be asynchronously pushed to the message platform via MQTT.

[0041] 2. Message layer: It includes the MQTT broker, which supports the MQTT and MQTTS protocols and is responsible for receiving messages from the acquisition layer for processing by the data processing layer; it also includes the Web Socket broker, which supports the WS and WSS protocols and receives messages from the data processing layer for use by the HMI screen of the external system.

[0042] In this specific embodiment, the message layer can utilize the open-source EMQX, a massively scalable, cloud-native, distributed IoT MQTT / WS messaging server. It provides efficient and reliable connectivity for massive IoT devices, enabling high-performance real-time movement and processing of message and event stream data, enabling the rapid construction of mission-critical IoT platforms and applications. It features open source, MQTT 5.0 support, massive connectivity, high performance, low latency, and high availability.

[0043] To further improve performance, the message layer adopts a distributed cluster architecture, which can handle a large number of clients and messages while ensuring high availability, fault tolerance and scalability.

[0044] 3. Cache layer: This layer provides two caches for the data processing layer, including Cache 1 and Cache 2. Cache 1 caches topics and payloads to prevent congestion caused by the data processing layer's delayed consumption of message layer data. Cache 2 caches business data to prevent congestion caused by delayed data storage at the data processing layer.

[0045] In a specific embodiment, the cache adopts memory cache, Redis cache, or Memcached cache.

[0046] To prevent memory exhaustion and ensure stable cache operation, the cache uses a First In, First Out (FIFO) purging strategy. This strategy determines which data to purge based on when it enters the cache. When cache space is insufficient, the oldest data entry is automatically deleted. In this specific embodiment, cache 1 is limited to 10,000 topic and payload messages, and cache 2 is limited to 1,000 business data items.

[0047] 4. Data processing layer: includes three modules: subscription module, which runs when subscription triggers; cache 1 processing module, which runs periodically; cache 2 processing module, which runs periodically.

[0048] The subscription module subscribes to MQTT data at the message layer. This includes two steps.

[0049] S411: Filter the data;

[0050] S412: Asynchronously storing the topic and payload data into cache 1 for valid data.

[0051] There are two main reasons for data filtering. First, due to the wildcard subscription used, some received topics are not required. Second, according to business logic, in situations such as unit shutdowns, no messages need to be received, reducing data pressure on the entire platform.

[0052] The cache 1 processing module obtains data from cache 1 and processes it, which includes four steps.

[0053] S421: Data analysis;

[0054] S422: Data tracking generates business data;

[0055] S423: asynchronously push the business data to the Web Socket broker of the message layer;

[0056] S424: Storing the business data in cache 2;

[0057] Data parsing refers to converting payload data into readable data, such as through JSON decomposition.

[0058] Data tracking means tracking and converting the original data. For example, the original data is a key-value pair of id and value, where id is in the format agreed upon in the configuration file or program. It needs to be converted into the format required by the business data. In some embodiments, some arithmetic operations need to be performed on the data.

[0059] The cache 2 processing module obtains business data from cache 2 and stores it in batches into the time series database of the storage layer. The specific steps include:

[0060] S431: Obtain cache 2 business data;

[0061] S432: Clear the business data obtained in cache 2;

[0062] S433: Batch storage to the time series database of the storage layer;

[0063] S434: If the storage is successful, the single process ends. If the storage fails, the business data that failed to be stored is stored again in cache 2.

[0064] In a specific embodiment, message push and cache storage can be completed quickly, while persistent storage of data is the process that affects efficiency the most, due to factors such as hard disk IO bottlenecks, network bottlenecks, etc. Batch storage can be used here to further improve storage efficiency.

[0065] Taking a certain embodiment as an example, it takes 0.3 seconds to collect a set of data, about 0 seconds to push the message and store it in the cache, and 0.5 seconds to store a set of data (including 0.3 seconds for connection processing and 0.2 seconds for data storage). At this time, data storage cannot keep up with the pace of data collection, which will cause system disconnection, data loss, etc. However, if four sets of data are saved at the same time after caching, it takes about 1.1 seconds (including 0.3 seconds for connection processing and 0.8 seconds for data storage). This is less than the 1.2 seconds it takes to collect four sets of data. This also illustrates the advantage of batch storage, that is, batch storage can reduce the resource overhead of the connection process and improve overall storage performance.

[0066] 5. Storage layer: This layer stores business data processed by the data processing layer. The storage layer is responsible for persistent storage of data to support subsequent data analysis and mining.

[0067] In a specific embodiment, the storage layer may use the open source InfluxDB time series database.

[0068] Alternatively, the storage layer can adopt existing distributed file systems, columnar databases, and other technologies, such as HDFS and Cassandra.

[0069] In order to further improve performance, the storage layer adopts a distributed cluster architecture to ensure storage performance and data security.

[0070] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A high-frequency industrial time series data storage system based on double caching, characterized by: It includes the collection layer, message layer, cache layer, data processing layer and storage layer from the bottom to the top; The collection layer is a platform that supports multiple data collection drivers. It is used to collect real-time data from equipment and sensors at industrial production sites and supports asynchronous push to the message layer through MQTT, providing a single source of industrial automation data to applications. The message layer adopts a distributed cluster architecture, including MQTT broker, and supports protocols including mqtt and mqtts, which is used to receive messages from the collection layer and send them to the data processing layer; the message layer also includes Web Socket broker, and supports protocols including ws and wss, which is used to receive messages from the data processing layer and send them to the HMI screen of the external system; The cache layer includes a first cache module and a second cache module, which are used to provide two caches for the data processing layer; the first cache module is a topic and payload cache, which is used to prevent the data processing layer from consuming message layer data in a timely manner and causing congestion; the second cache module is a business data cache, which is used to prevent the data processing layer from saving data in a timely manner and causing congestion; The storage layer includes a time series database.

2. According to claim 1, a high-frequency industrial time series data storage system based on double caching is characterized in that: The industrial communication protocols used in the acquisition layer include Modbus, OPC-UA or PLC-specific protocols; The equipment and sensors at industrial production sites are external systems including PLC, RTU, related instruments or third-party systems; The data pushed by the collection layer to the message layer is encoded in JSON format.

3. According to claim 1, a high-frequency industrial time series data storage system based on double caching is characterized in that: The message layer uses open source EMQX, which is an elastically scalable cloud-native distributed IoT MQTT / WS message server. It is used to connect multiple IoT devices, move and process messages and event stream data in real time, and build business-critical IoT platforms and applications.

4. According to claim 1, a high-frequency industrial time series data storage system based on double caching is characterized in that: The cache uses memory cache, Redis cache or Memcached cache; The cache uses a first-in-first-out (FIFO) cleaning strategy to clean up data based on the time when the data enters the cache. When there is insufficient space, the earliest data in the cache is automatically deleted.

5. According to claim 1, a high-frequency industrial time series data storage system based on double caching is characterized in that: The data processing layer includes a subscription module, a first cache processing module and a second cache processing module; the subscription module is triggered to run, and the first cache processing module and the second cache processing module are both run periodically; the subscription module is used to subscribe to the MQTT data of the message layer; the first cache processing module is used to obtain and process data from the first cache module; the second cache processing module is used to obtain business data from the second cache module and store it in batches into the time series database of the storage layer.

6. A high-frequency industrial time series data storage system based on double caching according to claim 5, characterized in that: The specific steps for the subscription module to subscribe to the MQTT data at the message layer are: S411: When using wildcard subscription, some received topics need to be removed, and when there is no need to receive messages according to business logic, the data is filtered; S412: Asynchronously storing the topic and payload data into the first cache module for the valid data.

7. The high-frequency industrial time series data storage system based on double caching according to claim 5 is characterized in that: The specific steps of the first cache processing module acquiring and processing data from the first cache module are: S421: Data parsing, converting payload data into readable data; S422: Data tracking, tracking and converting the original data into business data; S423: asynchronously push the business data to the Web Socket broker of the message layer; S424: Storing the business data in the second cache module.

8. The high-frequency industrial time series data storage system based on double caching according to claim 5 is characterized in that: The specific steps of the second cache processing module obtaining business data from the second cache module and storing it in batches into the time series database of the storage layer are as follows: S431: Acquire business data of the second cache module; S432: Clearing the business data acquired by the second cache module; S433: batch storage to the time series database of the storage layer; S434: If the storage is successful, the single process ends; If the storage fails, the business data that failed to be stored will be Stored in the second cache module.

9. The high-frequency industrial time series data storage system based on double caching according to claim 1 is characterized in that: The storage layer is used to persistently store business data processed by the data processing layer; The storage layer uses the InfluxDB time series database, distributed file system, or columnar database, including HDFS or Cassandra; The storage layer adopts a distributed cluster architecture to ensure storage performance and data security.

Images