Real-time data collection, processing and dump system based on Flink

Abstract

The real-time data collection, processing and dumping system based on Flink includes: a data collection module, which obtains indicator data and stores it in the Zabbix database; a data cache module, which builds a scheduled task, and then updates or supplements the indicator data in the Zabbix database through the scheduled task combined with the business database information, and stores it in the Redis cache; and a data dump module, in which Flink reads the cached indicator data, processes the cached indicator data according to the indicator diversion, and dumps it into the business database. A complete real-time data collection and dumping system is formed by the Zabbix database of the data collection module, the Redis cache of the data cache module, the Flink server of the data dump module, and the Kafka middleware of the data dump module. Through the diversion process of the data dump module and the setting of the water level trigger output, the parallelism of the entire real-time data collection and dumping system is improved in real time when the data information continues to increase, thereby alleviating data pressure and avoiding the monitoring system from not being able to work normally when the data pressure is too high, thereby improving system stability.

Description

Technical Field

[0001] The present invention belongs to the technical field of real-time data acquisition, processing and data storage, and in particular relates to a real-time data acquisition, processing and dumping system based on Flink. Background Art

[0002] In real-world business scenarios, there are demands for processing large amounts of data, complex data processing, and the need to dump partial information. Existing technologies lack scalability, stability, and flexibility. Furthermore, there is no comprehensive process for data collection, processing, and storage. Flink, a distributed streaming data processing framework that combines high throughput, low latency, and high performance, is often used in conjunction with Kafka to subscribe to and consume collected data. Redis cache offers high read and write performance and can be used as intermediate storage when processing real-time data, enhancing system stability.

[0003] For monitoring platforms, it is common for the system to acquire and process an excessive amount of data due to the excessive number of devices it monitors and manages. However, some existing real-time data processing processes have unstable defects. When the data pressure is too high, the monitoring system will not work properly. In addition, they often only focus on the processing of certain links and lack integrity and flexibility. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the present invention provides a real-time data acquisition, processing and dumping system based on Flink.

[0005] To achieve the above object, the present invention adopts the following technical solution: a real-time data acquisition, processing and dumping system based on Flink, comprising: a data acquisition module, which obtains indicator data and stores the indicator data in a Zabbix database;

[0006] The data cache module builds a scheduled task, and then updates or supplements the indicator data in the Zabbix database through the scheduled task combined with the business database information and stores it in the redis cache; and

[0007] The data dump module reads the cached indicator data and dumps the cached indicator data into the business database according to the indicator diversion process.

[0008] In order to optimize the above technical solution, the specific measures taken also include: obtaining the indicator data is specifically configuring a template for the collected indicator data, then associating the template of the collected indicator data with the Zabbix client to be monitored, and then collecting the indicator data of the monitored Zabbix client and storing it in the Zabbix database.

[0009] Furthermore, the data cache module includes a local server and a redis cache. The local server obtains indicator data from the Zabbix database and supplements it into the local business database to obtain complete indicator data. The redis cache is used to store the complete indicator data.

[0010] Furthermore, the scheduled task specifically synchronizes the indicator data in the Zabbix database periodically, obtains information in the local business database, combines the indicator data in the database with the information in the local business database to obtain complete indicator data, and then stores the complete indicator data in the redis cache.

[0011] Furthermore, the data dump module includes a flink server and a kafka middleware. The flink server reads the data pushed by kafka and the complete indicator data in the redis cache, and merges the data pushed by kafka with the complete indicator data in the redis cache. The kafka middleware is used to assist the flink server in consuming and processing the complete indicator data in the redis cache.

[0012] Furthermore, the data diversion is specifically to read the complete indicator data in the redis cache periodically through the flink server, and merge the complete indicator data in the timed redis cache with the information read from the kafka middleware, and then divert the merged complete indicator data according to the indicator type, and then add a random key to the merged complete indicator data, and divert the merged complete indicator data for a second time through the random key, and then configure the time window through the trigger, trigger the water level trigger output through the time window, and aggregate the merged complete indicator data into a list list and classify it into the business database.

[0013] Furthermore, the watermark trigger output specifically obtains A, B, X, and Y, and sets S. If the current system time is greater than both A-B+X+S and Y+S, the trigger data is forced to undergo a third diversion, and the complete fused indicator data after the third diversion is aggregated into a list, where A represents the earliest arriving event time within the timestamp, B represents the latest arriving event time within the timestamp, X represents the system time of the earliest arriving data within the timestamp, Y represents the system time of the latest arriving data within the timestamp, and S represents a delay time.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the Flink-based real-time data acquisition, processing, and dumping system according to any one of claims 1 to 7.

[0015] The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the real-time data acquisition, processing, and dumping system based on Flink as described in any one of claims 1 to 7 is implemented.

[0016] The beneficial effects of the present invention are:

[0017] 1. A complete real-time data collection and dumping system is formed by the Zabbix database of the data collection module, the Redis cache of the data cache module, the Flink server of the data dump module, and the Kafka middleware of the data dump module. By setting the watermark trigger output during the diversion process of the data dump module, the parallelism of the entire real-time data collection and dumping system is improved in real time when the data information continues to increase, thereby alleviating data pressure and avoiding the monitoring system from malfunctioning due to excessive data pressure, thereby improving system stability.

[0018] 2. The local service of the data cache module regularly accesses the Zabbix database and business database and stores them in the redis cache, making the system more flexible in processing data information.

[0019] 3. The indicator data obtained from the Zabbix database is supplemented into the local business database through the local server, so that the indicator data is adjusted in real time according to the information in the local business database, making the entire real-time data collection and dump system applicable to a variety of different business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of the real-time data acquisition, processing and dump system based on Flink proposed by the present invention;

[0021] Figure 2 This is a flow chart of the data acquisition module of the present invention;

[0022] Figure 3 This is a flow chart of the data cache module of the present invention;

[0023] Figure 4 This is a flow chart of the data dump module of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The embodiment of the present application provides a real-time data collection, processing and dumping system based on Flink, including: a data collection module, which obtains indicator data and stores the indicator data in a Zabbix database;

[0026] The data cache module builds a scheduled task, and then updates or supplements the indicator data in the Zabbix database through the scheduled task combined with the business database information and stores it in the redis cache; and

[0027] The data dump module reads the cached indicator data and dumps the cached indicator data into the business database through data diversion, which can realize the entire process from performance data collection to processing and then dumping it into the business database.

[0028] In Example 1, please refer to Figure 1 , the Zabbix database configures a template based on the collected indicator data, and associates the collected indicator data template with the Zabbix client that needs to be monitored, and then stores the indicator data collected from the Zabbix client into the Zabbix database, calls the required indicator data through the local server, and combines the obtained Zabbix data with the local business database in the local server to obtain complete indicator data, and then stores the complete indicator data in the redis cache. Then, the flink server reads the complete indicator data in the redis cache regularly, and merges the complete indicator data read from the redis cache with the information read from the kafka middleware. The fused complete indicator data is diverted according to the indicator type, and then a random key is added to the fused complete indicator data, and the fused complete indicator data is diverted for the second time through the random key. Finally, the time window is configured through the trigger, and the water level trigger output is triggered by the time window. The fused complete indicator data is aggregated into a list list and stored in the business database.

[0029] In this embodiment, the indicator data is device performance data. Supplementing the acquired Zabbix data with the local business database in the local server means modifying it by adding relevant notes or information such as the associated line name according to the indicator name and device number.

[0030] See also Figure 2 The data collection module is used to obtain indicator data. In the data collection module, the Zabbix database configures the template according to the indicator data, and associates the template with the Zabbix client that needs to be monitored. Then, the indicator data of the Zabbix client is collected through the Zabbix server or Zabbix proxy and stored in the redis cache.

[0031] In this embodiment, the original performance data in the Zabbix database needs to be processed and converted into platform business data.

[0032] See also Figure 3 When processing raw performance data, it is generally necessary to combine it with business data. The data cache module is mainly composed of local services and redis. Its main function is to build a scheduled task, periodically synchronize the collected data stored in the Zabbix database, and obtain information from the business database based on the raw performance data, combine the two into complete data, and write the assembled data into the redis cache. The specific process of building a scheduled task is to trigger the scheduled task, read the structured old indicator data from the redis cache, and then obtain new indicator data from the Zabbix database. Associate the Zabbix database with the local business database, supplement the corresponding fields, obtain complete indicator data, delete the invalid old indicator data in the redis cache, and supplement the complete indicator data.

[0033] In this embodiment, using redis cache as an intermediate step can not only provide good read and write performance, but also avoid direct impact on data collection and processing when accidents occur in the database.

[0034] See also Figure 3 The data dump module primarily consists of a Flink server and a Kafka database. After implementing data caching in the local service, the Flink operator sets an interval to read the complete indicator data from the Redis cache and merges it with the data pushed by the Kafka database. The merged indicator data is then distributed according to the indicator type, and the time period is set using the Flink time aggregation window. The merged indicator data within the specified time period is then calculated, and the watermark trigger output is set. A random key is then added to the merged complete indicator data, and the merged complete indicator data is split a second time using the random key. Finally, a time window is configured through the trigger, and the watermark trigger output is triggered by the time window. The merged complete indicator data is aggregated into a list and stored in the business database.

[0035] In this embodiment, when the amount of business data continues to expand, the pressure can be relieved by expanding the partition of the Kafka database. The Flink server can use the Kafka database as the data source, create a time window to perform business processing on the complete indicator data obtained, and store the result sink in the business database. It can also trigger the output by setting the watermark, so that when the amount of data continues to increase, the parallelism can be increased to alleviate the pressure and avoid data backlog in the time window of Flink.

[0036] Furthermore, when Flink creates a time window to process data, it can add a watermark trigger output. Specifically, the watermark trigger output obtains the earliest arriving event time A within the timestamp, the latest arriving event time B within the timestamp, the system time X of the earliest arriving data within the timestamp, and the system time Y of the latest arriving data within the timestamp. A delay time S is also defined for the system. The value of S needs to be determined based on the actual system operating environment to prevent the time spent on data processing from affecting the result judgment. When the current system time is simultaneously greater than (A-B+X+S) and (Y+S), the data is forced to be diverted for a third time. That is, the output is triggered when the interval between the arriving data timestamps (event time) and the system time (process time) both meet the timeout condition.

[0037] In another embodiment, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the Flink-based real-time data acquisition, processing, and dumping system as described in Example 1.

[0038] In another embodiment, the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the real-time data acquisition, processing, and dump system based on Flink as described in Example 1 is implemented.

[0039] In the embodiments disclosed herein, computer storage media can be tangible media that can contain or store programs for use by or in conjunction with an instruction execution system, device, or apparatus. Computer storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0040] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0041] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A real-time data collection, processing and dumping system based on Flink, characterized by: include: Data collection module, obtains indicator data and stores the indicator data in the Zabbix database; The data cache module builds a scheduled task, and then updates or supplements the indicator data in the Zabbix database through the scheduled task combined with the business database information and stores it in the redis cache; and The data dump module reads the cached indicator data and dumps the cached indicator data into the business database according to the indicator diversion process; The scheduled task specifically synchronizes the indicator data in the Zabbix database periodically, obtains information from the local business database, combines the indicator data in the database with the information in the local business database to obtain complete indicator data, and then stores the complete indicator data in the redis cache; The data dump module includes a flink server and a kafka middleware. The flink server reads the push data of the kafka middleware and the complete indicator data in the redis cache, and merges the push data of kafka with the complete indicator data in the redis cache. The kafka middleware is used to assist the flink server in consuming and processing the complete indicator data in the redis cache. The specific process of the diversion process is as follows: The Flink server periodically reads the complete indicator data from the Redis cache and merges the complete indicator data in the scheduled Redis cache with the information read from the Kafka database. The merged complete indicator data is then divided according to the indicator type and output is triggered by setting the watermark. A random key is then added to the merged complete indicator data, and the merged complete indicator data is divided a second time using the random key. The time window is configured through the trigger, and the watermark is triggered by the time window to trigger the output. The merged complete indicator data is aggregated into a list and stored in the business database. The watermark trigger output specifically obtains A, B, X, and Y, and sets S. If the current system time is greater than both A-B+X+S and Y+S, the data is forced to undergo a third diversion. The complete fused indicator data after the third diversion is aggregated into a list, where A represents the earliest event time within the timestamp, B represents the latest event time within the timestamp, X represents the system time of the earliest data within the timestamp, Y represents the system time of the latest data within the timestamp, and S represents a delay time.

2. The real-time data acquisition, processing and dumping system based on Flink according to claim 1 is characterized in that: Acquiring the indicator data specifically involves configuring a template for the collected indicator data, then associating the template for the collected indicator data with a Zabbix client to be monitored, then collecting the indicator data of the monitored Zabbix client, and storing it in a Zabbix database.

3. The real-time data acquisition, processing and dumping system based on Flink according to claim 1 is characterized in that: The data cache module includes a local server and a redis cache. The local server obtains indicator data from the Zabbix database and adds it to the local business database to obtain complete indicator data. The redis cache is used to store the complete indicator data.

4. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the Flink-based real-time data acquisition, processing and dumping system according to any one of claims 1 to 3.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the real-time data acquisition, processing, and dumping system based on Flink is implemented.

Images