Industrial real-time data management method, system, equipment and medium

Abstract

The present invention discloses an industrial real-time data management method, system, equipment and medium, and belongs to the field of Java / system technology. The technical problem to be solved by the present invention is how to improve data acquisition efficiency, optimize data storage performance and enhance data query efficiency. The technical scheme adopted is: data acquisition: real-time data is collected from a variety of industrial equipment and sensors, and the data is preprocessed and cached; data storage: a high-performance time series database is used to store data, and data partitioning and data compression are supported; data query: a fast data retrieval function is provided, and queries to establish data indexes and data caches are supported.

Description

Technical Field

[0001] The present invention relates to the field of Java / system technology, and in particular to an industrial real-time data management method, system, equipment and medium. Background Art

[0002] In the field of industrial automation and industrial Internet of Things, real-time data collection, storage and query is one of the key technologies. Existing industrial real-time data management systems usually face problems such as low data collection efficiency, poor data storage performance, and insufficient system scalability, which cannot meet the high requirements of modern industrial environments for data processing. Specifically:

[0003] ① Low data collection efficiency: When faced with a large number of data sources, the existing system has low data collection efficiency and cannot meet the needs of high-frequency data collection.

[0004] ②Poor data storage performance: The data storage module of the existing system has poor performance when processing large-scale data, which affects the real-time requirements.

[0005] ③ Low data reliability: When the network is unstable or the database is unavailable, the existing system is prone to data loss and cannot guarantee data reliability.

[0006] ④ High system complexity: The existing system has a complex architecture, which is difficult to maintain and manage, increasing operation and maintenance costs.

[0007] Therefore, how to improve data collection efficiency, optimize data storage performance and enhance data query efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0008] The technical task of the present invention is to provide an industrial real-time data management method, system, equipment and medium to solve the problem of how to improve data collection efficiency, optimize data storage performance and enhance data query efficiency.

[0009] The technical task of the present invention is achieved in the following way: a method for industrial real-time data management, the method is as follows:

[0010] Data collection: Efficiently collect real-time data from a variety of industrial equipment and sensors, and pre-process and cache the data;

[0011] Data storage: Use a high-performance time series database to store data, supporting data partitioning and data compression;

[0012] Data query: Provides fast data retrieval function and supports the establishment of data indexes and data cache queries.

[0013] As a preference, data collection is specifically as follows:

[0014] By writing data source adapters to communicate with different types of data source devices, multiple industrial protocols such as Modbus, OPC UA, MQTT, and HTTP are implemented. The plug-in modular design is adopted to dynamically load or unload data source adapters as needed, which is convenient for expansion and maintenance.

[0015] Set the collection cycle to read data from the data source device regularly or based on event triggers, and adjust the collection frequency through the configuration file or user interface to meet the needs of different application scenarios;

[0016] The collection engine supports multi-threaded parallel processing and can process multiple data source devices at the same time, improving the efficiency and throughput of data collection. It also performs pre-processing of the collected data such as deduplication, data cleaning, and format conversion to ensure data accuracy and consistency.

[0017] When the network is unstable or the database is unavailable, the data is temporarily stored in the memory to ensure that the data is not lost. The memory cache uses advanced data structures to support fast read and write operations. When the memory cache is full, the data is automatically transferred to the local persistent storage (such as disk) and written in batches when the network is restored or the database is available. At the same time, the cached data is compressed and stored using an efficient compression algorithm to reduce storage space.

[0018] Preferably, the data is stored as follows:

[0019] Data partitioning: Data is partitioned and stored according to time or other key fields to improve query efficiency. Data partitioning is configured according to specific business needs and supports dynamic adjustment.

[0020] Data compression: Use advanced compression algorithms (such as Gorilla, LZ4, etc.) to compress stored data to save storage space to the maximum extent without affecting reading performance.

[0021] Preferably, the data query is as follows:

[0022] Provide a unified data query API, supporting real-time and historical data queries;

[0023] Establish an efficient indexing mechanism through the data indexing layer to improve the speed of data retrieval;

[0024] The data cache layer uses a cache mechanism to cache hot data, reduce database query pressure, and improve query performance.

[0025] An industrial real-time data management system, the system comprising:

[0026] Data acquisition module, used to collect real-time data from various industrial equipment and sensors, and pre-process and cache the data;

[0027] The data storage module is used to use a high-performance time series database (such as KaiwuDB, InfluxDB, etc.) to store data persistently. The time series database can efficiently store and process large-scale time series data.

[0028] The data query module is used to provide fast data retrieval functions and support the establishment of data indexes and data cache queries.

[0029] Preferably, the data acquisition module comprises:

[0030] Data source adapter, used to communicate with different types of data source devices;

[0031] The collection engine is used to read data from the data source device regularly or based on event triggers and perform pre-processing;

[0032] The data preprocessing submodule is used to process multiple data source devices at the same time, improve the efficiency and throughput of data collection, and preprocess the collected data by deduplication, data cleaning, and format conversion to ensure the accuracy and consistency of the data;

[0033] The data cache unit is used to temporarily store data in memory when the network is unstable or the database is unavailable to ensure that the data is not lost. The memory cache uses advanced data structures to support fast read and write operations. When the memory cache is full, the data is automatically transferred to local persistent storage (such as disk) and written in batches when the network is restored or the database is available. An efficient compression algorithm is used to compress and store cached data to reduce storage space.

[0034] Preferably, the data storage module comprises:

[0035] The data writing layer is used to receive and process the collected data and write the collected data into the storage layer;

[0036] The data storage layer is used to use a time series database to persistently store data and supports data partitioning and data compression.

[0037] Preferably, the data query module includes:

[0038] The query API submodule is used to provide a unified data query API, supporting the query of real-time data and historical data;

[0039] Data indexing layer, used to establish an efficient indexing mechanism to improve the speed of data retrieval;

[0040] The data cache layer is used to adopt a cache mechanism to cache hot data, reduce the query pressure of the database, and improve query performance.

[0041] An electronic device comprising: a memory and at least one processor;

[0042] Wherein, the memory stores a computer program;

[0043] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the industrial real-time data management method as described above.

[0044] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the industrial real-time data management method as described above.

[0045] The industrial real-time data management method, system, device and medium of the present invention have the following advantages:

[0046] (1) The present invention supports multiple industrial protocols and improves the overall efficiency of data collection when facing a large number of data sources;

[0047] (ii) The data source acquisition module of the present invention improves the reliability of system acquisition through the data cache unit;

[0048] (III) The data storage module of the present invention adopts a time series database for storage, which greatly improves the data storage and processing performance; and adopts data compression to save storage space to the maximum extent;

[0049] (iv) The data query module of the present invention reduces the query pressure of hot data and improves the data response performance through a cache mechanism;

[0050] (V) The present invention improves data collection efficiency, optimizes data storage performance and enhances system data query efficiency, while reducing system complexity and system operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention is further described below in conjunction with the accompanying drawings.

[0052] Attached Figure 1 This is the architecture diagram of the data acquisition module;

[0053] Attached Figure 2 This is the architecture diagram of the data query module;

[0054] Attached Figure 3 This is the architecture diagram of the data query module. DETAILED DESCRIPTION

[0055] The industrial real-time data management method, system, device and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Embodiment 1:

[0057] This embodiment provides an industrial real-time data management method, which is specifically as follows:

[0058] S1. Data collection: Efficiently collect real-time data from a variety of industrial equipment and sensors, and pre-process and cache the data;

[0059] S2, Data storage: Use a high-performance time series database to store data, supporting data partitioning and data compression;

[0060] S3, data query: provides fast data retrieval function and supports the establishment of data index and data cache query.

[0061] The data collection in step S1 of this embodiment is specifically as follows:

[0062] S101. By writing data source adapters to communicate with different types of data source devices, multiple industrial protocols such as Modbus, OPC UA, MQTT, and HTTP are implemented. The plug-in modular design is adopted to dynamically load or unload the data source adapter as needed, which is convenient for expansion and maintenance.

[0063] S102, setting a collection cycle, reading data from a data source device regularly or according to event triggers, and adjusting the collection frequency through a configuration file or a user interface to meet the needs of different application scenarios;

[0064] S103. The collection engine supports multi-threaded parallel processing and can process multiple data source devices at the same time, thereby improving the efficiency and throughput of data collection, and pre-processing the collected data by deduplication, data cleaning, and format conversion to ensure the accuracy and consistency of the data;

[0065] S104. When the network is unstable or the database is unavailable, the data is temporarily stored in the memory to ensure that the data is not lost. The memory cache adopts an advanced data structure to support fast read and write operations. When the memory cache is full, the data is automatically transferred to the local persistent storage (such as a disk) and written in batches when the network is restored or the database is available. At the same time, the cached data is compressed and stored using an efficient compression algorithm to reduce storage space usage.

[0066] The data storage in step S2 of this embodiment is specifically as follows:

[0067] S201, data partitioning: partition and store data according to time or other key fields to improve query efficiency; data partitioning is configured according to specific business needs and supports dynamic adjustment;

[0068] S202. Data compression: Use advanced compression algorithms (such as Gorilla, LZ4, etc.) to compress the stored data to save storage space to the maximum extent without affecting the reading performance.

[0069] The data query in step S3 of this embodiment is specifically as follows:

[0070] S301. Provide a unified data query API to support real-time and historical data queries;

[0071] S302, establishing an efficient indexing mechanism through the data indexing layer to improve the speed of data retrieval;

[0072] S303. A cache mechanism is used through the data cache layer to cache hot data, reduce the query pressure of the database, and improve the query performance.

[0073] Embodiment 2:

[0074] This embodiment provides an industrial real-time data management system, the system comprising:

[0075] Data acquisition module, used to collect real-time data from various industrial equipment and sensors, and pre-process and cache the data;

[0076] The data storage module is used to use a high-performance time series database (such as KaiwuDB, InfluxDB, etc.) to store data persistently. The time series database can efficiently store and process large-scale time series data.

[0077] The data query module is used to provide fast data retrieval functions and support the establishment of data indexes and data cache queries.

[0078] As attached Figure 1 As shown, the data acquisition module includes:

[0079] Data source adapter, used to communicate with different types of data source devices;

[0080] The collection engine is used to read data from the data source device regularly or based on event triggers and perform pre-processing;

[0081] The data preprocessing submodule is used to process multiple data source devices at the same time, improve the efficiency and throughput of data collection, and preprocess the collected data by deduplication, data cleaning, and format conversion to ensure the accuracy and consistency of the data;

[0082] The data cache unit is used to temporarily store data in memory when the network is unstable or the database is unavailable to ensure that the data is not lost. The memory cache uses advanced data structures to support fast read and write operations. When the memory cache is full, the data is automatically transferred to local persistent storage (such as disk) and written in batches when the network is restored or the database is available. An efficient compression algorithm is used to compress and store cached data to reduce storage space.

[0083] As attached Figure 2 As shown, the data storage module includes:

[0084] The data writing layer is used to receive and process the collected data and write the collected data into the storage layer;

[0085] The data storage layer is used to use a time series database to persistently store data and supports data partitioning and data compression.

[0086] As attached Figure 3 As shown, the data query module includes:

[0087] The query API submodule is used to provide a unified data query API, supporting the query of real-time data and historical data;

[0088] Data indexing layer, used to establish an efficient indexing mechanism to improve the speed of data retrieval;

[0089] The data cache layer is used to adopt a cache mechanism to cache hot data, reduce the query pressure of the database, and improve query performance.

[0090] Embodiment 3:

[0091] This embodiment also provides an electronic device, including: a memory and a processor;

[0092] Wherein, the memory stores computer-executable instructions;

[0093] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the industrial real-time data management method in any embodiment of the present invention.

[0094] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0095] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0096] Embodiment 4:

[0097] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the industrial real-time data management method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0098] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0099] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0100] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0101] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An industrial real-time data management method, characterized in that: The method is as follows: Data collection: collect real-time data from a variety of industrial equipment and sensors, and pre-process and cache the data; Data storage: Use a high-performance time series database to store data, supporting data partitioning and data compression; Data query: Provides fast data retrieval function and supports the establishment of data indexes and data cache queries.

2. The industrial real-time data management method according to claim 1, characterized in that: The data collection details are as follows: By writing data source adapters to communicate with different types of data source devices, multiple industrial protocols such as Modbus, OPC UA, MQTT, and HTTP are implemented, and plug-in modular design is adopted to dynamically load or unload data source adapters as needed; Set the collection cycle to read data from the data source device regularly or based on event triggers, and adjust the collection frequency through the configuration file or user interface to meet the needs of different application scenarios; The collection engine supports multi-threaded parallel processing and can process multiple data source devices at the same time, improving the efficiency and throughput of data collection, and pre-processing the collected data by deduplication, data cleaning, and format conversion; When the network is unstable or the database is unavailable, the data is temporarily stored in the memory. The memory cache uses advanced data structures to support fast read and write operations. When the memory cache is full, the data is automatically transferred to the local persistent storage and written in batches when the network is restored or the database is available. At the same time, the cached data is compressed and stored using an efficient compression algorithm.

3. The industrial real-time data management method according to claim 1, characterized in that: The data storage is as follows: Data partitioning: Data is partitioned and stored according to time or other key fields. Data partitioning is configured according to specific business needs and supports dynamic adjustment. Data compression: Use advanced compression algorithms to compress stored data, saving storage space to the maximum extent without affecting reading performance.

4. The industrial real-time data management method according to any one of claims 1 to 3, characterized in that: The data query is as follows: Provide a unified data query API, supporting real-time and historical data queries; Establish an efficient indexing mechanism through the data indexing layer to improve the speed of data retrieval; The data cache layer uses a cache mechanism to cache hot data, reduce database query pressure, and improve query performance.

5. An industrial real-time data management system, characterized in that: The system includes: Data acquisition module, used to collect real-time data from various industrial equipment and sensors, and pre-process and cache the data; A data storage module is used to persistently store data using a high-performance time series database. The time series database can efficiently store and process large-scale time series data. The data query module is used to provide fast data retrieval functions and support the establishment of data indexes and data cache queries.

6. The industrial real-time data management system according to claim 5, characterized in that: The data acquisition module comprises: Data source adapter, used to communicate with different types of data source devices; The collection engine is used to read data from the data source device regularly or based on event triggers and perform pre-processing; The data preprocessing submodule is used to process multiple data source devices at the same time, improve the efficiency and throughput of data collection, and preprocess the collected data by deduplication, data cleaning, and format conversion to ensure the accuracy and consistency of the data; The data cache unit is used to temporarily store data in memory when the network is unstable or the database is unavailable to ensure that the data is not lost. The memory cache uses advanced data structures to support fast read and write operations. When the memory cache is full, the data is automatically transferred to the local persistent storage and written in batches when the network is restored or the database is available. An efficient compression algorithm is used to compress and store the cached data to reduce storage space.

7. The industrial real-time data management system according to claim 5, characterized in that: The data storage module comprises: The data writing layer is used to receive and process the collected data and write the collected data into the storage layer; The data storage layer is used to use a time series database to persistently store data and supports data partitioning and data compression.

8. The industrial real-time data management system according to any one of claims 5 to 7, characterized in that: The data query module includes: The query API submodule is used to provide a unified data query API, supporting the query of real-time data and historical data; Data indexing layer, used to establish an efficient indexing mechanism to improve the speed of data retrieval; The data cache layer is used to adopt a cache mechanism to cache hot data, reduce the query pressure of the database, and improve query performance.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the industrial real-time data management method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the industrial real-time data management method according to any one of claims 1 to 4.

Images