A task monitoring method, device, equipment and storage medium

Abstract

The present invention discloses a task monitoring method, device, equipment and storage medium, the method comprising: regularly traversing all pre-saved metric indicator instances, and obtaining the current value of each metric indicator instance each time all metric indicator instances are traversed; after obtaining the current value of each traversed metric indicator instance each time, uniformly packaging each metric indicator instance and the current value of each metric indicator instance to obtain corresponding task data; sending the task data to a pre-set database for storage, so as to enable monitoring of the corresponding Flink task by querying the data in the database; the value of any metric indicator instance is obtained by analyzing the indicator data of each metric indicator generated by the corresponding Flink task during operation. The present application can monitor the situation during the operation of the Flink task, so as to facilitate the corresponding statistical analysis and other operations when the entire Flink task situation needs to be understood.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and more specifically, to a task monitoring method, device, equipment and storage medium. Background Art

[0002] When using Flink as a processing engine and writing a Flink task to process data, data is generally extracted from multiple data sources. At the same time, for a certain data source, multiple degrees of parallelism are set to increase the speed of obtaining data from the current data source. In this case, different Flink tasks are basically distributed in different JVMs of different physical machines; and the metric statistics provided by Flink itself are the data of a single parallel task of the operator. When you want to know how much data the entire task receives or outputs, there is no way to perform corresponding statistical analysis. Summary of the invention

[0003] The purpose of the present invention is to provide a task monitoring method, device, equipment and storage medium, which can monitor the situation during the operation of a Flink task, so as to facilitate corresponding statistical analysis and other operations when it is necessary to understand the situation of the entire Flink task.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A task monitoring method, comprising:

[0006] Iterates through all pre-saved metric instances at regular intervals, and obtains the current value of each metric instance each time it traverses all metric instances. The value of any metric instance is obtained by analyzing the metric data of each metric generated by the corresponding Flink task during operation.

[0007] After obtaining the current value of each traversed metric instance each time, each metric instance and the current value of each metric instance are uniformly packaged to obtain the corresponding task data;

[0008] The task data is sent to a preset database for storage, so that the corresponding Flink task can be monitored by querying the data in the database.

[0009] Preferably, pre-saving the metric instance includes:

[0010] Determine the metrics that any Flink task needs to obtain corresponding metric data during its operation;

[0011] Based on each metric corresponding to the arbitrary Flink task, a corresponding metric instance is constructed and saved; wherein, each metric corresponding to the same metric instance is arranged in a manner such that the greater the possibility of being retrieved, the higher the order.

[0012] Preferably, constructing a metric instance includes:

[0013] Construct a metric instance corresponding to each metric of any Flink task in each Flink task, map each metric instance to different javabeans of the same type in a one-to-one correspondence, and store each metric corresponding to the mapped metric instance in each javabean.

[0014] Preferably, each metric instance and the current value of each metric instance are uniformly packaged to obtain corresponding task data, including:

[0015] Determine each metric instance as the current instance, and package the current time, each metric corresponding to the current instance, and the current value of the current instance into corresponding task data.

[0016] Preferably, sending the task data to a preset database for storage includes:

[0017] The task data is sent to the corresponding topic in Kafka, so that DruidIO can obtain the task data under the corresponding topic in Kafka.

[0018] Preferably, all pre-saved metric indicator instances are periodically traversed, including:

[0019] Use the Scheduled timer to periodically traverse all pre-saved metric instances.

[0020] Preferably, it also includes:

[0021] The DruidIO receives a query request, and displays the task data corresponding to the query request after drawing; wherein the query request is a request constructed in an SQL manner.

[0022] A task monitoring device, comprising:

[0023] The acquisition module is used to: regularly traverse all pre-saved metric indicator instances, and obtain the current value of each metric indicator instance each time all metric indicator instances are traversed; wherein, the value of any metric indicator instance is obtained by analyzing the indicator data of each metric indicator generated during the operation of the corresponding Flink task;

[0024] The packaging module is used to: after obtaining the current value of each metric indicator instance each time, uniformly package each metric indicator instance and the current value of each metric indicator instance to obtain corresponding task data;

[0025] The storage module is used to send the task data to a preset database for storage, so as to monitor the corresponding Flink task by querying the data in the database.

[0026] A task monitoring device, comprising:

[0027] Memory for storing computer programs;

[0028] A processor is used to implement the steps of the task monitoring method as described in any one of the above items when executing the computer program.

[0029] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above task monitoring methods are implemented.

[0030] The present invention provides a task monitoring method, device, equipment and storage medium, the method comprising: regularly traversing all pre-saved metric indicator instances, and obtaining the current value of each metric indicator instance each time all metric indicator instances are traversed; after obtaining the current value of each traversed metric indicator instance each time, uniformly packaging each metric indicator instance and the current value of each metric indicator instance to obtain corresponding task data; sending the task data to a pre-set database for storage, so as to enable monitoring of the corresponding Flink task by querying the data in the database; the value of any metric indicator instance is obtained by analyzing the indicator data of each metric indicator generated by the corresponding Flink task during operation. The present application obtains the value of the metric indicator instance corresponding to each Flink task by regularly traversing, obtains the result of the indicator data analysis of each metric indicator generated by each Flink task during operation, and then uniformly packages these results and stores them in the database, so as to perform statistical analysis and other operations on the situation of the operation process of each Flink task. It can be seen that the present application can monitor the situation during the operation of the Flink task, so as to facilitate the corresponding statistical analysis and other operations when the situation of the entire Flink task needs to be understood. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0032] Figure 1 A flowchart of a task monitoring method provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a data processing flow in a task monitoring method provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the structure of a task monitoring device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1 , which shows a flow chart of a task monitoring method provided by an embodiment of the present invention, which may specifically include:

[0037] S11: Periodically traverse all pre-saved metric indicator instances, and obtain the current value of each metric indicator instance each time all metric indicator instances are traversed; wherein the value of any metric indicator instance is obtained by analyzing the indicator data of each metric indicator generated during the operation of the corresponding Flink task.

[0038] It should be noted that Flink is a distributed real-time stream processing engine. It is a real-time stream computing framework that achieves low latency, high throughput, and high performance through the Dataflow streaming computing mode. At the same time, Flink supports highly fault-tolerant state management to prevent the state from being lost during the calculation process due to system anomalies. Flink periodically implements state persistence maintenance through distributed snapshot technology Checkpoints, so that correct results can be calculated even in the case of system downtime or anomalies.

[0039] In the embodiment of the present application, Flink is used as a processing engine, and a Flink task is written to implement the corresponding data processing; and, in the embodiment of the present application, a custom Reporter is set in Flink, and then a task monitoring solution is implemented based on the custom Reporter. Specifically, the present application can pre-set a metric instance, and in the process of running any Flink task, the indicator data of each metric generated during the running of the arbitrary Flink task can be obtained in real time or regularly, and then statistical analysis is performed based on the obtained indicator data generated during the running of the arbitrary Flink task, and the value of the metric instance corresponding to the arbitrary Flink task is obtained; wherein, the metric is an indicator representing various situations during the running of the Flink task, such as source type, running host, source type name, source type ID, parallelism ID, etc., the indicator data is the value of the obtained metric, and the value of the metric instance is the value obtained by analyzing the indicator data, such as the amount of received data, the amount of sent data and the system load, etc.

[0040] S12: After obtaining the current value of each traversed metric instance each time, each metric instance and the current value of each metric instance are uniformly packaged to obtain corresponding task data.

[0041] In order to facilitate statistical analysis of data, the embodiments of the present application can uniformly package the current values ​​of each metric instance obtained in the current traversal to obtain corresponding task data after each traversal to the current value of each metric instance; and the specific method of uniformly packaging the current values ​​of each metric instance can be determined according to actual needs, such as converting them all into a unified format, etc., which is within the scope of protection of the present invention.

[0042] S13: Send the task data to a preset database for storage, so that the corresponding Flink task can be monitored by querying the data in the database.

[0043] After the corresponding task data is obtained through unified packaging each time, the task data can be sent to a pre-set database for storage, so that relevant personnel can query the task data from the database and perform corresponding statistical analysis and other operations.

[0044] This application obtains the value of the metric instance corresponding to each Flink task through timed traversal, obtains the results of the metric data analysis of each metric generated by each Flink task during operation, and then stores these results in a database after unified packaging, so as to perform statistical analysis and other operations on the operation process of each Flink task. It can be seen that this application can monitor the situation during the operation of the Flink task, so as to facilitate the corresponding statistical analysis and other operations when it is necessary to understand the situation of the entire Flink task.

[0045] An embodiment of the present invention provides a task monitoring method, which pre-saves a measurement indicator instance, and specifically may include:

[0046] Determine the metrics that any Flink task needs to obtain corresponding metric data during its operation;

[0047] Build and save corresponding metric instances based on the metric indicators corresponding to the arbitrary Flink task; wherein the metric indicators corresponding to the same metric instance are arranged in a way that the greater the possibility of being retrieved, the higher the priority;

[0048] Among them, constructing a metric instance may include:

[0049] Construct a metric instance corresponding to each metric of any Flink task in each Flink task, map each metric instance to different javabeans of the same type in a one-to-one correspondence, and store each metric corresponding to the mapped metric instance in each javabean;

[0050] The timed traversal of all pre-saved metric instances may include:

[0051] Use the Scheduled timer to periodically traverse all pre-saved metric instances.

[0052] It should be noted that the embodiment of the present application sets the combination method of the metric name in the Reporter. Specifically, the specific name of the metric in Flink is relatively complex, which will be determined according to the source type, running host, source type name, source type ID, parallelism ID, etc.; and the metric will be grouped according to the source type, which includes JobManager, TaskManager, operator, etc. In order to facilitate the filtering and identification of metric names during query, the embodiment of the present application can arrange the metric names accordingly in a way that the greater the possibility of being retrieved, the higher the front, so that when customizing, in the main configuration file of Flink (flink-conf.yaml), add an independent combination method and modify the metric name format of the operator to:

[0053] metrics.scope.operator:

[0054] <host>.operator.<tm_ID> .<job_ID> .<task_ID> .<task_attempt_ID> .<task_attempt_num> .<operator_ID> .<subtask_index> .<operator_name> ;

[0055] The attributes in the angle brackets are all metrics, which will be automatically filled in according to the actual situation when the corresponding Flink task is running. In this way, the metrics are sorted and can be placed at the end when the metric name is strange or contains a lot of uncertain characters, so that the clearer and more specific metric names (that is, the most likely names to be retrieved) can be easily obtained. Then the combination of metrics is used as a parameter to inform the custom Reporter, which is convenient for separating metrics in the program:

[0056] metrics.reporter.Kafka.metrics.scope.operator:

[0057] <host>.operator.<tm_ID> .<job_ID> .<task_ID> .<task_attempt_ID> .<task_attempt_num> .<operator_ID> .<subtask_index> .<operator_name> .

[0058] The embodiment of the present application can also perform data standardization in a custom Reporter, mainly by implementing MetricReporter to build a custom Fafka Reporter, and implementing a Scheduled timer to regularly push out the indicator data of the metric indicator; specifically, the embodiment of the present application can save different types of metric indicator instances in memory, such as Counters (counts), Gauges (measurements), Histograms (statistics), and Meters (rates), etc. When the number of metric indicator instances is modified, the mapping relationship of the metric indicator instances is synchronously adjusted, and different metric instances are mapped to different JavaBeans of the same type. In addition, when constructing a JavaBean, the embodiment of the present application defines the fields of concern (i.e., metrics) in the JavaBean definition, such as the host name (host), task ID (jobID), task name (jobName), subtask index (subtaskIndex) of the task, operator ID (operatorID), operator name (operatorName), metric name (key), and data table name (tableName), etc., and then when mapping to generate the JavaBean, the metric combination is implemented according to the above-defined format; for example, the operator name (operatorName) can be taken out, if it starts with "Source:", then it can be determined that the JavaBean is a data source operator, and after marking, the relevant data can be assigned and saved.

[0059] A task monitoring method provided by an embodiment of the present invention sends task data to a preset database for storage, which may include:

[0060] Send the task data to the corresponding topic in Kafka so that DruidIO can obtain the task data under the corresponding topic in Kafka;

[0061] In addition, each metric instance and the current value of each metric instance are uniformly packaged to obtain corresponding task data, which may include:

[0062] Determine each metric instance as the current instance, and package the current time, each metric corresponding to the current instance, and the current value of the current instance into the corresponding task data;

[0063] In addition, a task monitoring method provided by an embodiment of the present invention may further include:

[0064] DruidIO receives a query request and displays the task data corresponding to the query request after drawing; wherein the query request is a request constructed in SQL mode.

[0065] When implementing standardized data storage, the embodiment of the present application can be configured to send out the indicator data of the real metric every 3 seconds based on the mapping relationship between the customized metric instance and the JavaBean and the Scheduled interface of the timer. Specifically, in the timed execution method, all saved metric instances are traversed. For each metric instance, because there is a JavaBean associated with it, and the basic information of the metric instance (that is, the various metric indicators corresponding to the metric instance) is saved in the JavaBean, a unified data sending format can be constructed, which includes three parts: 1: current time, 2: JavaBean basic information, 3: current value of the metric instance; and then all the metric information of the current time is uniformly packaged.

[0066] The embodiment of the present application can build a Kafka data transmitter, and send the prepared data to the topic created and specified in advance under Kafka through the Kafka transmitter. The topic uniformly receives the indicator data of all Flink clusters, and uses the json data format to realize the storage of the received data. The embodiment of the present application can also build DruidIO, and obtain the data in the above-mentioned topic under Kafka through the built-in Kafka plug-in of DruidIO. At the same time, a table for storing task data is defined, and the conversion method of task data is defined, and the field relationship mapping is performed on the extracted task data, and the data in json format is mapped into a single-layer column. DruidIO uniformly provides "time" (time), "host" (host name), "jobID" (task ID), "operatorID" (operator ID), "tableName" (data table name), "sourceOperator" (whether it is a data source operator), "key" (metric name), "value" (metric value) and other fields available for query, and DruidIO provides a table for querying storage in the form of SQL, but it is necessary to encapsulate it once using http request. Specifically, you can perform data aggregation queries on DruidIO according to business needs. For example, if you need to query the total data input trend chart of a task within a certain period of time, you can build a SQL query idea - filter the task ID to the task ID of interest, filter the metric name to the "numRecords" of interest, filter the data to the data source operator, and filter the time to the specified time period. Finally, through the aggregation function, the total data value received by each data source at different time points can be counted. The sample SQL can be as follows:

[0067]

[0068] The above SQL query outputs the total value of input data corresponding to different time points (divided by different input tables). The example can be as follows: [

[0070] {time:1,value:100,tableName:t1},

[0071] {time:1,value:50,tableName:t2},

[0072] {time:1,value:200,tableName:t3},

[0073] {time:2,value:200,tableName:t1},

[0074] {time:2,value:100,tableName:t2},

[0075] {time:2,value:400,tableName:t3},

[0076] {time:3,value:300,tableName:t1},

[0077] {time:3,value:150,tableName:t2},

[0078] {time:3,value:800,tableName:t3} ]

[0080] By plotting the above data, you can view the total amount of data at different time points, and you can also view the total amount of specific source tables. In this way, you can obtain data on various metrics that are more worthy of attention. In addition, the embodiment of the present application can also directly store the task data in clickhouse, mySQL, etc. after obtaining the task data, which is within the scope of protection of the present invention.

[0081] The data flow processing diagram of this application can be as follows Figure 2 As shown, specifically, this application first customizes Flink Reporter, classifies and standardizes various metrics of Flink tasks in Reporter, and stores metric-related data in the database. Finally, aggregate queries are used to count the overall traffic of Flink tasks. This application can split and reorganize metric-related data in a specific way in Reporter, and add some additional specific data and identification information to complete the standardization of various metrics. After the data is pushed to a specific middleware or storage, the analysis and presentation of Flink task metrics from the overall perspective of the Flink task can be added. In this way, the metric statistics query of the entire task can be performed in a relatively simple way, and it supports queries on past metric data.

[0082] The embodiment of the present invention also provides a task monitoring device, such as Figure 3 As shown, it may include:

[0083] The acquisition module 11 is used to: regularly traverse all pre-saved metric indicator instances, and obtain the current value of each metric indicator instance each time all metric indicator instances are traversed; wherein the value of any metric indicator instance is obtained by analyzing the indicator data of each metric indicator generated during the operation of the corresponding Flink task;

[0084] The packaging module 12 is used to: after obtaining the current value of each metric indicator instance each time, uniformly package each metric indicator instance and the current value of each metric indicator instance to obtain corresponding task data;

[0085] The storage module 13 is used to send the task data to a preset database for storage, so as to monitor the corresponding Flink task by querying the data in the database.

[0086] The embodiment of the present invention further provides a task monitoring device, which may include:

[0087] Memory for storing computer programs;

[0088] A processor is used to implement the steps of any of the above task monitoring methods when executing a computer program.

[0089] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above task monitoring methods can be implemented.

[0090] It should be noted that the description of the relevant parts of a task monitoring device, equipment and storage medium provided by the embodiment of the present invention can be found in the detailed description of the corresponding parts of a task monitoring method provided by the embodiment of the present invention, which will not be repeated here. In addition, the parts of the above technical solutions provided by the embodiment of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0091] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / host> < / host>

Claims

1. A task monitoring method, characterized in that: include: Iterates through all pre-saved metric instances at regular intervals, and obtains the current value of each metric instance each time it traverses all metric instances; the value of any metric instance is obtained by analyzing the metric data of each metric generated by the corresponding Flink task during operation; After obtaining the current value of each traversed metric instance each time, each metric instance and the current value of each metric instance are uniformly packaged to obtain the corresponding task data; Send the task data to a preset database for storage, so as to enable monitoring of the corresponding Flink task by querying the data in the database; Pre-saving the measurement indicator instance includes: Determine the metrics that any Flink task needs to obtain corresponding metric data during its operation; Build and save corresponding metric instances based on the metric indicators corresponding to the arbitrary Flink task; wherein the metric indicators corresponding to the same metric instance are arranged in a way that the greater the possibility of being retrieved, the higher the priority; Among them, constructing a measurement indicator instance includes: Construct a metric instance corresponding to each metric of any Flink task in each Flink task, map each metric instance to different javabeans of the same type in a one-to-one correspondence, and store each metric corresponding to the mapped metric instance in each javabean.

2. The method according to claim 1, characterized in that Each metric instance and its current value are packaged together to obtain the corresponding task data, including: Determine each metric instance as the current instance, and package the current time, each metric corresponding to the current instance, and the current value of the current instance into corresponding task data.

3. The method according to claim 2, characterized in that Sending the task data to a preset database for storage includes: The task data is sent to the corresponding topic in Kafka, so that DruidIO can obtain the task data under the corresponding topic in Kafka.

4. The method according to claim 3, characterized in that Regularly traverse all pre-saved metric instances, including: Use the Scheduled timer to periodically traverse all pre-saved metric instances.

5. The method according to claim 4, characterized in that Also includes: The DruidIO receives a query request, and displays the task data corresponding to the query request after drawing; wherein the query request is a request constructed in an SQL manner.

6. A task monitoring device, characterized in that: include: The acquisition module is used to: regularly traverse all pre-saved metric indicator instances, and obtain the current value of each metric indicator instance each time all metric indicator instances are traversed; wherein, the value of any metric indicator instance is obtained by analyzing the indicator data of each metric indicator generated during the operation of the corresponding Flink task; The packaging module is used to: after obtaining the current value of each metric indicator instance each time, uniformly package each metric indicator instance and the current value of each metric indicator instance to obtain corresponding task data; A storage module is used to: send the task data to a preset database for storage, so as to monitor the corresponding Flink task by querying the data in the database; Pre-saving the measurement indicator instance includes: Determine the metrics that any Flink task needs to obtain corresponding metric data during its operation; Build and save corresponding metric instances based on the metric indicators corresponding to the arbitrary Flink task; wherein the metric indicators corresponding to the same metric instance are arranged in a way that the greater the possibility of being retrieved, the higher the priority; Among them, constructing a measurement indicator instance includes: Construct a metric instance corresponding to each metric of any Flink task in each Flink task, map each metric instance to different javabeans of the same type in a one-to-one correspondence, and store each metric corresponding to the mapped metric instance in each javabean.

7. A task monitoring device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the task monitoring method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task monitoring method according to any one of claims 1 to 5 are implemented.

Images