Novel intelligent data acquisition platform based on distributed architecture

Abstract

The invention provides a novel intelligent data acquisition platform based on a distributed architecture, and belongs to the technical field of industrial data acquisition and real-time data processing.The novel intelligent data acquisition platform based on the distributed architecture is constructed, and a system adopts a three-level optimization strategy: firstly, an asynchronous processing mechanism is constructed based on a Redis message queue; request flow peak clipping and resource dynamic scheduling are realized, and the system throughput is significantly improved. Secondly, through a multi-level cache architecture design (Redis + memory database), an intelligent data prefetching mechanism is established, and the core service response time delay is reduced to a millisecond level; and finally, a TDengine time sequence database cluster is innovatively adopted, and the concurrent processing capability of data is improved in combination with the characteristics of column storage and distributed architecture of the TDengine time sequence database cluster. The system avalanche risk existing in a traditional architecture is effectively solved, and a reliable localization technical base is provided for real-time data acquisition.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial data acquisition and real-time data processing, and particularly to a new intelligent data acquisition platform based on a distributed architecture. Background Art

[0002] In the field of industrial data acquisition and real-time monitoring, when facing large-scale real-time data acquisition scenarios, traditional data acquisition architectures often face the following technical pain points:

[0003] High concurrency performance bottleneck: In the data writing scenario, traditional centralized data acquisition architectures (such as architectures based on MySQL or a single message queue) are limited by the I / O throughput capacity of a single node and the lock competition mechanism, and are prone to request backlogs, sharp increases in response latency, and even service crashes.

[0004] Vulnerability of the data link: Traditional architectures rely on a single communication link and synchronous processing mechanisms. In complex industrial network environments (such as bandwidth fluctuations and device disconnections), it is easy to cause data loss or duplicate transmissions, and it is difficult to meet the stringent requirements for data integrity in industrial scenarios.

[0005] Lack of system disaster tolerance: Existing solutions lack dynamic resource scheduling and fast fault recovery mechanisms in the event of node failures or sudden increases in load, and there is a risk of cascading failures (avalanche effect), resulting in business interruptions (such as production line downtime).

[0006] Asynchronous architecture based on Kafka: Although it supports high throughput, it relies on disk persistence and is difficult to meet the requirements of industrial scenarios; Memory database caching solutions (such as a single Redis node): High risk of cache breakdown, and unable to effectively handle the uneven distribution of hot and cold data; Traditional time series databases: Weak distributed expansion ability, significant decline in write performance as the data volume increases, and high operation and maintenance costs. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention provides a new intelligent data acquisition platform based on a distributed architecture. Aiming at the high concurrency, low latency, and high reliability requirements of large-scale real-time data acquisition in industrial scenarios, through multi-level collaborative optimization technology, it solves problems such as the system performance bottleneck, insufficient data link stability, and business continuity risks of traditional architectures, providing domestic technical support for industrial automation.

[0008] The technical solution of the present invention is:

[0009] A new intelligent data acquisition platform based on a distributed architecture,

[0010] including:

[0011] The data acquisition layer is responsible for the acquisition and transmission of real-time data;

[0012] The data processing layer is responsible for the asynchronous processing and caching of data;

[0013] The data storage layer is responsible for the persistent storage and query of data;

[0014] A three - level optimization strategy is adopted: an asynchronous processing mechanism, a multi - level cache architecture design, and a time - series database cluster.

[0015] Furthermore,

[0016] Based on the Redis message queue, an asynchronous processing mechanism is constructed to achieve peak shaving of request traffic and dynamic resource scheduling, improving the system throughput;

[0017] Moreover,

[0018] The data acquisition layer sends the acquired data to the Redis message queue, and the data processing layer asynchronously obtains the data from the queue for processing;

[0019] Through a dynamic scheduling algorithm, the resource allocation is dynamically adjusted according to the system load condition.

[0020] Moreover,

[0021] Through the multi - level cache architecture design, an intelligent data pre - fetching mechanism is established to reduce the business response latency to the millisecond level;

[0022] Moreover,

[0023] The multi - level cache architecture design combines Redis and an in - memory database. The intelligent pre - fetching mechanism pre - loads data into the cache in advance according to business requirements; the cached data is dynamically adjusted according to the access frequency.

[0024] Furthermore,

[0025] The TDengine time - series database cluster is adopted. Combining its column - based storage and distributed architecture features, the data concurrent processing ability is improved.

[0026] Moreover,

[0027] The database cluster ensures the high availability of the system and the security of data through a load - balancing and fail - over mechanism.

[0028] Moreover,

[0029] The data acquisition work is completed by the end - gateway. The data is formatted at the end - gateway and transmitted to the system in real - time through MQTT.

[0030] The beneficial effects of the present invention are

[0031] The present invention constructs a new intelligent data acquisition platform based on a distributed architecture and adopts a three-level optimization strategy, effectively solving the key problems faced by traditional data acquisition architectures in large-scale real-time data acquisition scenarios, such as insufficient high-concurrency processing capabilities, poor data link stability, and difficulty in ensuring business continuity. The system throughput has been significantly improved, the response latency of core services has been reduced, and the data concurrency processing ability has been greatly enhanced, providing a reliable domestic technology foundation for real-time data acquisition. Detailed implementation manners

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] The present invention provides a new intelligent data acquisition platform based on a distributed architecture. By constructing a new intelligent data acquisition platform based on a distributed architecture, the system adopts a three-level optimization strategy: First, an asynchronous processing mechanism is constructed based on the Redis message queue to achieve request traffic peak shaving and dynamic resource scheduling, significantly improving the system throughput; Second, through the design of a multi-level cache architecture (Redis + in-memory database), an intelligent data prefetch mechanism is established to reduce the core service response latency to the millisecond level; Finally, the TDengine time-series database cluster is innovatively adopted, combined with its columnar storage and distributed architecture characteristics, to improve the data concurrency processing ability. It effectively solves the system avalanche risk existing in the traditional architecture and provides a reliable domestic technology foundation for real-time data acquisition.

[0034] 1. Data acquisition: Users can dynamically configure device acquisition points through the gateway acquisition program, and collect device operation data in real time and upload it northward to the system platform; With the development of 5G communication technology, 5G provides a new technical solution for data acquisition. Through the characteristics of large bandwidth and low latency of 5G, this solution ensures the real-time and accuracy of data acquisition. The data acquisition work is completed by the end gateway, and the data is formatted at the end gateway and transmitted to the system in real time through MQTT.

[0035] 2. Redis message queue: By introducing the Redis message queue, the system implements an asynchronous processing mechanism, effectively solving the problem of request accumulation in high-concurrency scenarios, significantly improving the system throughput, and being able to dynamically adjust resource allocation according to the real-time load, avoiding resource waste and system overload, and further improving the system stability and efficiency.

[0036] 3. Multi-level Cache Architecture: Combining Redis and in-memory databases, the system constructs an efficient multi-level cache architecture, significantly reducing data access latency. By analyzing data access patterns, the system can prefetch frequently accessed data in advance, significantly reducing the response latency of core services and meeting the stringent requirements of real-time monitoring scenarios.

[0037] 4. Time Series Database Cluster: The system innovatively adopts the TDengine time series database cluster, making full use of its columnar storage and distributed architecture features to achieve efficient data processing. The introduction of TDengine not only improves data storage efficiency but also significantly reduces storage costs. At the same time, it supports fast query and analysis, providing strong data support for real-time monitoring.

[0038] The specific solutions are as follows:

[0039] 1. System Architecture:

[0040] The system adopts a distributed architecture, including a data collection layer, a data processing layer, and a data storage layer.

[0041] The data collection layer is responsible for the collection and transmission of real-time data. The data processing layer is responsible for the asynchronous processing and caching of data. The data storage layer is responsible for the persistent storage and query of data.

[0042] 2. Asynchronous Processing Mechanism:

[0043] The data collection layer sends the collected data to the Redis message queue, and the data processing layer asynchronously fetches the data from the queue for processing.

[0044] Through a dynamic scheduling algorithm, the resource allocation is dynamically adjusted according to the system load situation to ensure the stability of the system in high-concurrency scenarios.

[0045] 3. Multi-level Cache Architecture:

[0046] The data processing layer adopts a multi-level cache architecture that combines Redis and in-memory databases. The intelligent prefetch mechanism preloads data into the cache in advance according to business requirements.

[0047] The cached data is dynamically adjusted according to the access frequency to ensure fast access to high-frequency data.

[0048] 4. Time Series Database Cluster:

[0049] The data storage layer adopts the TDengine time series database cluster, leveraging its columnar storage and distributed architecture features to achieve efficient data storage and query.

[0050] The database cluster ensures the high availability of the system and the security of data through a load balancing and failover mechanism.

[0051] The above are only the preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A new intelligent data acquisition platform based on a distributed architecture, characterized in that it includes: The data acquisition layer is responsible for the acquisition and transmission of real-time data; The data processing layer is responsible for the asynchronous processing and caching of data; The data storage layer is responsible for the persistent storage and query of data; It adopts a three-level optimization strategy: asynchronous processing mechanism, multi-level cache architecture design, and time-series database cluster.

2. The platform according to claim 1, characterized in that Based on the Redis message queue, an asynchronous processing mechanism is built to achieve request traffic peak shaving and dynamic resource scheduling, and improve the system throughput.

3. The platform according to claim 2, characterized in that The data acquisition layer sends the acquired data to the Redis message queue, and the data processing layer asynchronously obtains the data from the queue for processing; Through a dynamic scheduling algorithm, the resource allocation is dynamically adjusted according to the system load condition.

4. The platform according to claim 3, characterized in that Through the multi-level cache architecture design, an intelligent data prefetch mechanism is established to reduce the business response delay to the millisecond level.

5. The platform according to claim 4, characterized in that The multi-level cache architecture design combines Redis and an in-memory database, and the intelligent prefetch mechanism preloads data into the cache in advance according to business requirements; the cached data is dynamically adjusted according to the access frequency.

6. The platform according to claim 1, characterized in that It adopts a TDengine time-series database cluster, combined with its columnar storage and distributed architecture characteristics, to improve the data concurrent processing ability.

7. The platform according to claim 6, characterized in that The database cluster ensures the high availability of the system and the security of data through a load balancing and failover mechanism.

8. The platform according to claim 1, characterized in that The data acquisition work is completed by the end gateway, and the data is formatted at the end gateway and transmitted to the system in real time through MQTT.