Dynamic monitoring method for Flink on YARN operation

Abstract

The invention relates to the technical field of big data, and discloses a Flink on YARN job dynamic monitoring method, which comprises the following steps: for any node, judging whether the node runs a YARN node manager or not; if yes, a target container executor in the node is determined, and the target container executor is used for Flink operation; for any target container executor, determining a sub-process list and a process identifier of each sub-process in the sub-process list; determining a network port monitored by each sub-process based on the process identifier of each sub-process; and obtaining target monitoring data based on the network port monitored by each sub-process. Compared with a traditional Pushgateway scheme, the method has the advantages that the problem of single-point performance load in a monitoring system is reduced, the stability is improved, the complexity of manual configuration of monitoring information is reduced, efficient and dynamic monitoring of Flink operation is realized, and the accuracy and the real-time performance of the monitoring process are ensured.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a dynamic monitoring method for a Flink on YARN job. Background Art

[0002] In today's big data and real-time data processing space, Apache Flink has become a popular tool for supporting data stream analysis and batch processing workloads. However, with the increasing amount of data and the diversification of data processing requirements, deploying and managing Flink jobs on large-scale clusters has become more complex. The Flink on YARN mode allows dynamic allocation and recycling of resources, ensuring that Flink jobs can fully utilize available computing and storage resources for efficient data processing.

[0003] In Flink on YARN mode, the dynamic distribution and scheduling of Flink job processes TaskManager (task manager) and JobManager (job manager) pose a challenge to monitoring and data collection. This is because they may be randomly scheduled to run on YARN cluster nodes, and multiple JobManagers and TaskManagers may run on a node. The exposed monitoring port cannot be determined, and the address and monitoring port of TaskManager and JobManager cannot be directly obtained.

[0004] Currently, the existing Flink on YARN job monitoring solution mainly relies on the Pushgateway component, which requires its address to be explicitly specified in the Flink job configuration and continuously pushes monitoring data to the Pushgateway during the job running process. However, the existing solution requires operators running Flink jobs to manually configure the address of the Pushgateway component, which increases the complexity of deployment and use, and requires additional information about the monitoring system. Secondly, the centralized architecture of Pushgateway makes it a potential single point of failure in the cluster. If Pushgateway fails or has performance problems, it will affect the stability of the entire monitoring system. In summary, the existing monitoring solution has the problems of cumbersome configuration and single point of failure risks, and is not suitable for large-scale and dynamic Flink on YARN cluster environments. Summary of the invention

[0005] In view of this, the present invention provides a dynamic monitoring method for Flink on YARN jobs to solve the problems of complicated configuration and single point failure risk in existing monitoring solutions.

[0006] In a first aspect, the present invention provides a method for dynamically monitoring a Flink on YARN job, the method comprising:

[0007] For any node, determine whether the node is running a YARN node manager;

[0008] If the node is running a YARN node manager, determine at least one target container executor in the node, which is used to perform the Flink job;

[0009] For any target container executor, determine a subprocess list of the target container executor and a process identifier of each subprocess in the subprocess list, wherein the subprocess list includes subprocesses of a task manager and a job manager running on the node;

[0010] Based on the process ID of each child process, determine the network port that each child process listens on;

[0011] Get the target monitoring data based on the network port monitored by each child process.

[0012] The dynamic monitoring method of Flink on YARN jobs provided by an embodiment of the present invention confirms whether a YARN node manager is running on a node. When the node belongs to a YARN cluster, the target container executor for the Flink job in the node is determined, and then a list of subprocesses of the target container executor is determined, including subprocesses of the task manager and the job manager running on the node, so as to ensure that the monitoring covers all subprocesses related to the Flink job and provide a comprehensive monitoring perspective. Then, based on the process identifier of each subprocess, the network port monitored by each subprocess is determined, and the target monitoring data is obtained through the information monitored by each network port. Compared with the traditional Pushgateway solution, the single-point performance load problem in the monitoring system is reduced, the stability is improved, and the tediousness of manually configuring monitoring information is reduced, so as to realize efficient dynamic monitoring of Flink jobs and ensure the accuracy and real-time performance of the monitoring process.

[0013] In an optional implementation, when the node runs a YARN node manager, determining at least one target container executor in the node includes:

[0014] If the node is running a YARN node manager, determine the node management process corresponding to the YARN node manager of the node;

[0015] Get all container executors of the node management process;

[0016] At least one target container executor of the Flink type is determined from all container executors of the node management process.

[0017] The dynamic monitoring method of Flink on YARN jobs provided by the embodiment of the present invention clearly identifies the node management process corresponding to the YARN node manager on the node, thereby obtaining all container executors of the node management process, and screening out the target container executor of the Flink type, thereby ensuring that the monitoring covers all containers that may run Flink jobs, thereby improving the comprehensiveness and completeness of the monitoring.

[0018] In an optional embodiment, the method further includes:

[0019] Get the application ID corresponding to each target container executor.

[0020] The dynamic monitoring method of the Flink on YARN job provided in the embodiment of the present invention can associate the subsequently acquired monitoring data with the specific application by obtaining the application identifier corresponding to each target container executor, thereby realizing application-level monitoring, which helps to better understand and analyze the running status of a specific application.

[0021] In an optional implementation, obtaining target monitoring data based on the network port monitored by each child process includes:

[0022] Based on any network port, obtain the monitoring data of the network port;

[0023] Determine whether the monitoring data of the network port conforms to the monitoring data format;

[0024] When the monitoring data conforms to the monitoring data format, the monitoring data is added to the target monitoring data.

[0025] The dynamic monitoring method of the Flink on YARN job provided by the embodiment of the present invention obtains the monitoring data of each network port, determines whether the monitoring data conforms to the monitoring data format, and screens the monitoring data to ensure that the final target monitoring data is valid and reliable.

[0026] In an optional implementation, after determining whether the monitoring data of the network port conforms to the monitoring data format, the method further includes:

[0027] When the monitoring data does not conform to the monitoring data format, the monitoring data of the next network port is continuously judged.

[0028] The dynamic monitoring method for Flink on YARN jobs provided by an embodiment of the present invention continues to judge the monitoring data of the next network port when the monitoring data does not conform to the monitoring data format, because the monitoring data may not be the desired monitoring data, to ensure that the final target monitoring data is valid and reliable.

[0029] In an optional embodiment, the method further includes:

[0030] The process identifier corresponding to each subprocess is associated with the application identifier corresponding to the subprocess.

[0031] The dynamic monitoring method for Flink on YARN jobs provided by the embodiment of the present invention can form a complete monitoring data link by associating the monitoring data of each subprocess with its corresponding application program identifier, and can perform refined management of each process of each application.

[0032] In a second aspect, the present invention provides a dynamic monitoring device for a Flink on YARN job, the device comprising:

[0033] A judgment module is used to judge whether a YARN node manager is running on any node.

[0034] A first determination module is used to determine at least one target container executor in a node when a YARN node manager is running on the node, where the target container executor is used to perform a Flink job;

[0035] A second determination module is used to determine, for any target container executor, a subprocess list of the target container executor and a process identifier of each subprocess in the subprocess list, wherein the subprocess list includes subprocesses of a task manager and a job manager running on the node;

[0036] A third determination module is used to determine the network port monitored by each sub-process based on the process identifier of each sub-process;

[0037] The monitoring module is used to obtain target monitoring data based on the network port monitored by each subprocess.

[0038] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the dynamic monitoring method of the Flink on YARN job of the above-mentioned first aspect or any corresponding embodiment thereof.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for dynamically monitoring Flink on YARN jobs of the first aspect or any corresponding embodiment thereof.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, where the computer instructions are used to enable a computer to execute the dynamic monitoring method of a Flink on YARN job according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flowchart of a dynamic monitoring method for a Flink on YARN job according to an embodiment of the present invention;

[0043] Figure 2 is a flowchart of another method for dynamic monitoring of Flink on YARN jobs according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a monitoring terminal according to an embodiment of the present invention;

[0045] Figure 4 is a flowchart of another method for dynamic monitoring of Flink on YARN jobs according to an embodiment of the present invention;

[0046] Figure 5 is a structural block diagram of a dynamic monitoring device for a Flink on YARN job according to an embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] The existing Flink on YARN job monitoring solution mainly relies on the Pushgateway component, requiring operators running Flink jobs to manually configure the address of the Pushgateway component, which increases the complexity of deployment and use, and requires additional information about the monitoring system. Secondly, the centralized architecture of Pushgateway makes it a potential single point of failure in the cluster. If Pushgateway fails or has performance problems, it will affect the stability of the entire monitoring system. The dynamic monitoring method for Flink on YARN jobs provided in an embodiment of the present invention improves stability, realizes efficient dynamic monitoring of Flink jobs, and ensures the accuracy and real-time performance of the monitoring process.

[0050] According to an embodiment of the present invention, an embodiment of a dynamic monitoring method for a Flink on YARN job is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] This embodiment provides a dynamic monitoring method for Flink on YARN jobs, which can be used to monitor terminals. Figure 1 is a flowchart of a dynamic monitoring method for a Flink on YARN job according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0052] Step S101, for any node, determine whether the node is running a YARN node manager. Specifically, since there are a large number of nodes in the host, and different nodes may be located in different clusters, the embodiment of the present invention is used to obtain monitoring data when Flink onYARN mode is running, so it is necessary to determine whether the node belongs to the YARN cluster, that is, to determine whether the node is running a YARN node manager (YARN NodeManager). By checking the process list on the node to find out whether there is a YARN node manager process, you can also query the node's status information through YARN's API (Application Programming Interface) to determine whether the node is registered as a YARN node manager.

[0053] Step S102, when the node is running with a YARN node manager, at least one target container executor in the node is determined, and the target container executor is used to perform Flink jobs. Specifically, when the node is running with a YARN node manager, that is, the node belongs to a YARN cluster, since there are multiple container executors (Container-Executor) on the node for executing different processes, it is necessary to filter out the container executor of the Flink type from all container executors to ensure that the monitoring focus is on the container that actually runs the Flink job, thereby improving the monitoring efficiency.

[0054] Step S103, for any target container executor, determine the sub-process list of the target container executor and the process ID of each sub-process in the sub-process list, the sub-process list includes the sub-processes of the task manager and the job manager running on the node. Specifically, when the Flink job is submitted to the YARN cluster, the YARN node manager will start the target container executor according to the needs of the job, and these target container executors will further start the task manager (TaskManager) and the job manager (JobManager) to perform specific job tasks. Among them, the job manager is responsible for the coordination and management of Flink jobs, and the task manager is responsible for the execution of specific tasks. Therefore, through the above-mentioned process relationship, the sub-process list of the target container executor and the process ID of each sub-process can be obtained, thereby realizing detailed monitoring and management of Flink jobs.

[0055] Step S104, based on the process ID of each sub-process, determine the network port that each sub-process listens to. Specifically, each task manager process or each job manager process may listen to multiple network ports. For example, Flink's job manager usually listens to port 8081 to provide a web interface and API interface, while the task manager may listen to port 9090 to provide task status information. Therefore, existing tools, such as netstat (Network Statistics), can be used to accurately locate the network port that each sub-process listens to based on the process ID of each sub-process, so as to solve the problem that multiple job managers and task managers may run on a node and the exposed monitoring port cannot be determined.

[0056] Step S105, based on the network port monitored by each child process, the target monitoring data is obtained. Specifically, according to the network port monitored by each child process, it is determined from which network ports the monitoring data can be obtained, and the target monitoring data is obtained from the specified network port by using HTTP (Hypertext Transfer Protocol) request or other protocols, and then the target monitoring data can be parsed to obtain the relevant data of Flink operation on the YARN cluster, such as CPU (Central Processing Unit) usage, memory usage, task execution status, etc., to achieve efficient dynamic monitoring of Flink operation on the YARN cluster.

[0057] The dynamic monitoring method of Flink on YARN jobs provided by an embodiment of the present invention confirms whether a YARN node manager is running on a node. When the node belongs to a YARN cluster, the target container executor for the Flink job in the node is determined, and then a list of subprocesses of the target container executor is determined, including subprocesses of the task manager and the job manager running on the node, so as to ensure that the monitoring covers all subprocesses related to the Flink job and provide a comprehensive monitoring perspective. Then, based on the process identifier of each subprocess, the network port monitored by each subprocess is determined, and the target monitoring data is obtained through the information monitored by each network port. Compared with the traditional Pushgateway solution, the single-point performance load problem in the monitoring system is reduced, the stability is improved, and the tediousness of manually configuring monitoring information is reduced, so as to realize efficient dynamic monitoring of Flink jobs and ensure the accuracy and real-time performance of the monitoring process.

[0058] In this embodiment, a dynamic monitoring method for a Flink on YARN job is provided, which can be used for the above-mentioned monitoring terminal. Figure 2FIG. 4 is a flowchart of another method for dynamically monitoring a Flink on YARN job according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0059] Step S201: for any node, determine whether the node is running a YARN node manager. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0060] Step S202: When the node runs a YARN node manager, determine at least one target container executor in the node, where the target container executor is used to perform a Flink job.

[0061] Specifically, the above step S202 includes:

[0062] Step S2021, when the node runs a YARN node manager, determine the node management process corresponding to the YARN node manager of the node. Specifically, when the node runs a YARN node manager, it means that the node has a node management process for executing Flink job tasks.

[0063] Step S2022, obtain all container executors of the node management process. Specifically, obtain all container executors managed by the node management process, and these container executors are used to execute various tasks.

[0064] Step S2023, at least one target container executor of the Flink type is determined from all container executors of the node management process. Specifically, by checking the environment variables and startup commands of the container executor, the target container executor of the Flink type is screened out, ensuring that the Flink job running on the YARN cluster can be accurately identified and monitored.

[0065] Step S203, obtain the application identifier corresponding to each target container executor. Specifically, since different target container executors perform different tasks and may serve different applications, obtaining the application identifier corresponding to each target container executor can associate the monitoring data obtained subsequently with the specific application, realize application-level monitoring, and help better understand and analyze the running status of a specific application.

[0066] Step S204: for any target container executor, determine the subprocess list of the target container executor and the process identifier of each subprocess in the subprocess list, where the subprocess list includes the subprocesses of the task manager and the job manager running on the node. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0067] Step S205: Based on the process ID of each child process, determine the network port that each child process monitors. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0068] Step S206, obtaining target monitoring data based on the network port monitored by each subprocess.

[0069] Specifically, the above step S206 includes:

[0070] Step S2061, based on any network port, obtain the monitoring data of the network port. Specifically, since each task manager or job manager may monitor multiple network ports, but not all network ports need to be monitored, it is necessary to judge the monitoring data. First, obtain the monitoring data of any network port.

[0071] Step S2062, determine whether the monitoring data of the network port conforms to the monitoring data format. Specifically, the operator can predefine the format of the monitoring data, such as data type, field name, field value range, etc., to confirm whether the acquired monitoring data conforms to the predefined monitoring data format, so as to decide whether to include it in the monitoring.

[0072] Step S2063: if the monitoring data conforms to the monitoring data format, the monitoring data is added to the target monitoring data. Specifically, if the monitoring data conforms to the monitoring data format, it is considered that the monitoring data needs to be monitored, and the monitoring data is added to the target monitoring data.

[0073] Step S2064, when the monitored data does not conform to the monitoring data format, continue to judge the monitored data of the next network port. Specifically, if the monitored data does not conform to the monitoring data format, the monitored data does not need to be monitored, and then continue to judge the monitored data of the next network port until the monitoring data of all network ports are judged and the final target monitoring data is obtained, ensuring that the data of the final target monitoring data is valid and reliable, thereby improving the accuracy of monitoring.

[0074] Step S207, associate the monitoring data corresponding to each subprocess with the application program ID corresponding to the subprocess. Specifically, when associating, the process ID of the subprocess can be associated with the application program ID, thereby realizing the association of the monitoring data with the application program, which is convenient for monitoring and analyzing the operation of the application program.

[0075] In some optional embodiments, Figure 3 is a schematic diagram of a monitoring terminal according to an embodiment of the present invention. Figure 3As shown, the indicator collection module on the monitoring terminal is used to collect the target monitoring data through the above steps, the indicator cache module is used to add the target monitoring data to the cache record, and a unified interface for exposing indicators to the outside is provided to facilitate other devices to collect monitoring data.

[0076] In some optional embodiments, Figure 4 FIG. 4 is a flowchart of another method for dynamically monitoring a Flink on YARN job according to an embodiment of the present invention. Figure 4 As shown, for any node, first determine whether there is a YARN node manager running on the node, so as to determine whether there is a YARN node management process on the node. If not, continue to determine the next node. If it exists, obtain all container executors of the YARN node management process. Then, filter out the target container executor of Flink type from all container executors. Then, for each target container executor, determine the child process list and the process ID of the child process of the target container executor. For any child process, determine the network port that the child process listens to based on the process ID of the child process. For any network port, obtain the monitoring data of the network port. Then, determine whether the monitoring data conforms to the monitoring data format. If it conforms, add the monitoring data to the target monitoring data. If it does not conform, continue to determine the next network port until all network ports of all child processes are determined, and the final target monitoring data is obtained.

[0077] The dynamic monitoring method of Flink on YARN jobs provided by an embodiment of the present invention confirms whether a YARN node manager is running on a node. When the node belongs to a YARN cluster, the target container executor for the Flink job in the node is determined, and then a list of subprocesses of the target container executor is determined, including subprocesses of the task manager and the job manager running on the node, so as to ensure that the monitoring covers all subprocesses related to the Flink job and provide a comprehensive monitoring perspective. Then, based on the process identifier of each subprocess, the network port monitored by each subprocess is determined, and the target monitoring data is obtained through the information monitored by each network port. Compared with the traditional Pushgateway solution, the single-point performance load problem in the monitoring system is reduced, the stability is improved, and the tediousness of manually configuring monitoring information is reduced, so as to realize efficient dynamic monitoring of Flink jobs and ensure the accuracy and real-time performance of the monitoring process.

[0078] In this embodiment, a dynamic monitoring device for Flink on YARN jobs is also provided, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0079] This embodiment provides a dynamic monitoring device for Flink on YARN jobs, such as Figure 5 As shown, including:

[0080] The judgment module 501 is used to judge, for any node, whether the node is running a YARN node manager.

[0081] The first determination module 502 is used to determine at least one target container executor in the node when the node runs a YARN node manager, and the target container executor is used to perform a Flink job.

[0082] The second determination module 503 is used to determine, for any target container executor, a subprocess list of the target container executor and a process identifier of each subprocess in the subprocess list, wherein the subprocess list includes subprocesses of a task manager and a job manager running on the node.

[0083] The third determination module 504 is used to determine the network port monitored by each sub-process based on the process identifier of each sub-process.

[0084] The monitoring module 505 is used to obtain target monitoring data based on the network port monitored by each sub-process.

[0085] In some optional implementations, the first determining module 502 includes:

[0086] The first determining unit is used to determine the node management process corresponding to the YARN node manager of the node when the node runs the YARN node manager.

[0087] The first acquisition unit is used to acquire all container executors of the node management process.

[0088] The second determining unit is used to determine at least one target container executor of the Flink type from all container executors of the node management process.

[0089] In some optional embodiments, the device further comprises:

[0090] The acquisition module is used to obtain the application program identifier corresponding to each target container executor.

[0091] In some optional implementations, the monitoring module 505 includes:

[0092] The second acquisition unit is used to acquire monitoring data of the network port based on any network port.

[0093] The judging unit is used to judge whether the monitoring data of the network port conforms to the monitoring data format.

[0094] The third determining unit is configured to add the monitoring data to the target monitoring data if the monitoring data conforms to the monitoring data format.

[0095] In some optional implementations, after the determination unit, the device further includes:

[0096] The monitoring judgment module is used to continue to judge the monitoring data of the next network port when the monitoring data does not conform to the monitoring data format.

[0097] In some optional embodiments, the device further comprises:

[0098] The association module is used to associate the process identifier corresponding to each sub-process with the application identifier corresponding to the sub-process.

[0099] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0100] The dynamic monitoring device of the Flink on YARN job in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0101] The embodiment of the present invention also provides a computer device having the above Figure 5 The following figure shows the dynamic monitoring setup for a Flink on YARN job.

[0102] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0103] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0104] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0105] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0107] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0108] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0109] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0110] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0111] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A dynamic monitoring method for Flink on YARN jobs, characterized in that: The method comprises: For any node, determine whether the node is running a YARN node manager; In a case where the node runs a YARN node manager, determining at least one target container executor in the node, the target container executor being used to perform a Flink job; For any target container executor, determine a subprocess list of the target container executor and a process identifier of each subprocess in the subprocess list, wherein the subprocess list includes subprocesses of a task manager and a job manager running on the node; Based on the process ID of each child process, determine the network port that each child process listens on; Get the target monitoring data based on the network port monitored by each child process.

2. The method according to claim 1, characterized in that In the case where the node runs a YARN node manager, determining at least one target container executor in the node includes: In the case where the node runs a YARN node manager, determining a node management process corresponding to the YARN node manager of the node; Obtain all container executors of the node management process; At least one target container executor of the Flink type is determined from all container executors of the node management process.

3. The method according to claim 1, characterized in that The method further comprises: Get the application ID corresponding to each target container executor.

4. The method according to claim 3, characterized in that: The method of obtaining target monitoring data based on the network port monitored by each subprocess includes: Based on any network port, obtaining monitoring data of the network port; Determine whether the monitoring data of the network port conforms to the monitoring data format; In the case where the monitoring data conforms to the monitoring data format, the monitoring data is added to the target monitoring data.

5. The method according to claim 4, characterized in that After determining whether the monitoring data of the network port conforms to the monitoring data format, the method further includes: When the monitoring data does not conform to the monitoring data format, the monitoring data of the next network port is continuously judged.

6. The method according to claim 3, characterized in that The method further comprises: The process identifier corresponding to each sub-process is associated with the application identifier corresponding to the sub-process.

7. A dynamic monitoring device for Flink on YARN jobs, characterized in that: The device comprises: A judgment module, used for judging, for any node, whether the node is running a YARN node manager; A first determination module is used to determine at least one target container executor in the node when the node runs a YARN node manager, and the target container executor is used to perform a Flink job; A second determination module is used to determine, for any target container executor, a subprocess list of the target container executor and a process identifier of each subprocess in the subprocess list, wherein the subprocess list includes subprocesses of a task manager and a job manager running on the node; A third determination module is used to determine the network port monitored by each sub-process based on the process identifier of each sub-process; The monitoring module is used to obtain target monitoring data based on the network port monitored by each subprocess.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the dynamic monitoring method for the Flink on YARN job according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the dynamic monitoring method of a Flink on YARN job according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the dynamic monitoring method of a Flink on YARN job according to any one of claims 1 to 6.

Images