A task monitoring method, apparatus, and device

Abstract

This application discloses a task monitoring method, apparatus, and device for refining the monitoring levels of the Spark distributed computing system. The method is applied to the Spark execution end, which communicates with the Spark driver. The method includes: monitoring each method in each task based on a preset monitoring type to obtain monitoring values; generating monitoring data based on the monitoring object name, monitoring type name, and monitoring values; concatenating the monitoring data belonging to the same task to obtain a target string; and sending a data packet generated based on the target string to the driver, enabling the driver to monitor each method in the execution end's task based on the data packet. Therefore, this application can monitor different methods under the same task, thus refining the monitoring levels of the Spark engine. Furthermore, using strings for data communication ensures compatibility with most data types.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a task monitoring method, apparatus, and device. Background Technology

[0002] Spark is currently the most common and popular computing engine in the field of big data distributed computing. Unlike Hadoop's MapReduce (MR) which writes intermediate results to disk each time, Spark caches data in memory, significantly improving the performance of iterative computations. It provides a variety of computation operators and supports the standard Structured Query Language (SQL), making big data programming more flexible and efficient for developers. As an excellent open-source distributed computing framework, Spark also has its own monitoring system, which greatly benefits its testability, performance optimization, operational assessment, and data statistics. Spark's metrics system is further built upon third-party metrics repositories and mainly includes: the data source representing the metric, which can be the application's status, Java Virtual Machine (JVM) information, Spark's execution end information, etc.; the instance name representing the metric system, i.e., the monitored object, which can be the driver or execution end; and the final aggregation point for the metric data, allowing Spark to send monitoring data to the console or external monitoring services.

[0003] Spark's monitoring level reaches the task level, but in reality, a user may define multiple operators in a single task.

[0004] For example, in iterating through partitions (Foreach Partition) or mapping partitions (Map Partition), users may perform various complex operations on the data in each partition. Furthermore, if there is data skew in the partitions, the execution time will also be different. If technical personnel need to understand the execution status of operators within these partitions, such as the execution time of the operators, this will be crucial.

[0005] Therefore, how to refine the monitoring levels of the Spark engine is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a task monitoring method, apparatus, and device to address the problem of how to refine the monitoring levels of the Spark engine.

[0007] In a first aspect, embodiments of this application provide a task monitoring method applied to the execution end of a distributed computing system Spark, wherein the execution end and the Spark driver are communicatively connected, and the method includes:

[0008] Each method in each task is monitored based on a preset monitoring type to obtain monitoring values, and monitoring data is generated based on the monitoring object name, monitoring type name, and the monitoring values.

[0009] By concatenating the monitoring data belonging to the same task, the target string is obtained;

[0010] A data packet generated based on the target string is sent to the driver, so that the driver can listen to each method in the task of the execution end based on the data packet.

[0011] The execution end of this application generates monitoring data based on the monitoring object name, monitoring type name, and monitoring value. For example, the execution end monitors method 01 based on the time consumption type, obtains the monitoring value of 64 milliseconds, and generates monitoring data: method 01, time consumption, 0046ms.

[0012] Furthermore, the monitoring data of the same task are concatenated, and the string formed by concatenating multiple methods, namely the target string, is sent to the driver in the form of a data packet so that the driver can listen to the information in the target string.

[0013] Therefore, based on the above technical solution, various methods under a task can be monitored, thereby refining the monitoring level of the Spark engine in the distributed computing system.

[0014] In one possible implementation, generating monitoring data based on the monitored object name, the monitoring type name, and the monitoring value includes:

[0015] The monitoring data is generated by concatenating the monitoring object name, the monitoring type name, and the monitoring value into a string.

[0016] Continuing with the example of monitoring method 01 based on the execution time type and obtaining a monitoring value of 64 milliseconds, we can choose "-" as the separator to concatenate method 01, time, and 0046ms to generate monitoring data method 01-time-0046ms. In this way, both the monitoring data and the target string output by concatenating the monitoring data are of string type, which is more convenient for compatibility with most data types.

[0017] In one possible implementation, the data packet is a heartbeat packet;

[0018] Sending the data packet generated based on the target string to the driver includes:

[0019] The heartbeat packet is sent to the driver at a preset period.

[0020] Unlike the communication packets used to transmit information after the execution end and the driver end establish a communication connection, the heartbeat packet is used to confirm the working status of the execution end. This application can use the heartbeat packet to communicate the target string data. In addition to the target string, the heartbeat packet can also include execution end status information, such as the execution end status indicators, etc.

[0021] Secondly, embodiments of this application provide a task monitoring method applied to the driver of Spark, wherein the driver and the execution end of Spark are communicatively connected, and the method includes:

[0022] The execution terminal receives data packets sent by the execution terminal; wherein the data packets are generated by the execution terminal based on a target string, the target string is generated by the execution terminal by concatenating various monitoring data belonging to the same task, the monitoring data is generated by the execution terminal based on the monitoring object name, monitoring type name and monitoring value, and the monitoring value is obtained by the execution terminal by monitoring each method in each task based on a preset monitoring type;

[0023] The data packet is parsed to obtain the target string;

[0024] If the monitoring value in the target string is updated, indicator data is generated based on the target string corresponding to the updated monitoring value, and the indicator data is sent to the monitoring system.

[0025] Based on the above technical solution, the driver can listen to the monitoring value in the target string. For example, the driver can listen to the monitoring value in the target string through a listener. If the monitoring value is updated, the driver can generate indicator data according to the target string corresponding to the updated monitoring value and send the indicator data to the monitoring system. The monitoring system can be a console or an externally connected visual monitoring system. The generated indicator data can be provided to the monitoring system for further monitoring or display.

[0026] In one possible implementation, the data packet is a heartbeat packet;

[0027] The step of parsing the data packet to obtain the target string includes:

[0028] The heartbeat packet is parsed to obtain the target string.

[0029] In practice, the heartbeat receiver on the driver side can receive and parse the heartbeat packets, and after scheduling and dispatching, send the target string to the module on the driver side for listening.

[0030] Thirdly, embodiments of this application provide a task monitoring device applied to the execution end of Spark, wherein the execution end and the Spark driver are communicatively connected, and the device includes:

[0031] The first generation module is used to monitor each method in each task based on a preset monitoring type, obtain monitoring values, and generate monitoring data based on the monitoring object name, monitoring type name, and the monitoring values.

[0032] The splicing module is used to splice together various monitoring data belonging to the same task to obtain the target string;

[0033] The sending module sends a data packet generated based on the target string to the driver, so that the driver can listen to each method in the task of the execution end based on the data packet.

[0034] In one possible implementation, the first generation module is specifically used for:

[0035] The monitoring data is generated by concatenating the monitoring object name, the monitoring type name, and the monitoring value into a string.

[0036] Fourthly, embodiments of this application provide a task monitoring device applied to the Spark driver, wherein the driver and the Spark execution end are communicatively connected, and the device includes:

[0037] A receiving module is used to receive data packets sent by the execution end; wherein the data packets are generated by the execution end based on a target string, the target string is generated by the execution end by concatenating various monitoring data belonging to the same task, the monitoring data is generated by the execution end based on the monitoring object name, monitoring type name and monitoring value, and the monitoring value is obtained by the execution end by monitoring each method in each task based on a preset monitoring type;

[0038] The parsing module is used to parse the data packet to obtain the target string;

[0039] The second generation module is used to generate indicator data based on the target string corresponding to the updated monitoring value if the monitoring value in the target string is updated, and then send the indicator data to the monitoring system.

[0040] Fifthly, embodiments of this application provide a computer device, the device including a processor and a memory;

[0041] The memory is used to store computer-executable programs;

[0042] The processor is configured to execute the computer-executable program, causing the method described in either the first aspect or the second aspect to be performed.

[0043] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer-executable program, which, when invoked by a computer device, causes the computer device to perform the method described in any one of the first or second aspects. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a task monitoring method provided in an embodiment of this application;

[0046] Figure 2 This is a schematic flowchart illustrating a task monitoring method provided in an embodiment of this application.

[0047] Figure 3 This application provides a schematic diagram of the structure of an execution-end task monitoring device.

[0048] Figure 4 A schematic diagram of a driver-side task monitoring device provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] To facilitate understanding of the embodiments of this application, the terms used in the embodiments of this application will be explained below:

[0052] I. Spark Engine

[0053] Spark is currently the most common and popular computing engine in the field of big data distributed computing. Unlike Hadoop's MapReduce (MR) which writes intermediate results to disk each time, Spark caches data in memory, significantly improving the performance of iterative computations. It provides a variety of computation operators and supports the standard Structured Query Language (SQL), making big data programming more flexible and efficient for developers.

[0054] As an excellent open-source distributed computing framework, Spark also has its own monitoring system, which greatly benefits its testability, performance optimization, operation and maintenance assessment, and data statistics. Spark's metrics system is further built on third-party metrics repositories and mainly includes: the data source representing the metrics, which can be the application status, Java Virtual Machine (JVM) information, Spark execution information, etc.; the instance name representing the metrics system, i.e., the monitored object, which can be the driver or execution end, etc.; and the final aggregation point for the metrics data, where Spark can send monitoring data to the console or external monitoring services.

[0055] II. Event Bus (ListenerBus)

[0056] ListenerBus refers to the trait that allows a device to receive listening events and send them to the corresponding listener.

[0057] ListenerBus mainly includes methods for maintaining all registered listeners, adding or removing listeners, delivering events to all listeners, delivering events to a specified listener, and finding a list of listeners of the same type.

[0058] III. Driver

[0059] The driver refers to the driver node in Spark, which is used to run the main method in a Spark program and execute the actual code.

[0060] The main functions of the driver include: converting user programs into jobs, enabling task scheduling between execution ends, monitoring the execution status of execution ends, and displaying the running status through the user interface.

[0061] IV. Executor

[0062] An execution node in Spark refers to a node that executes specific tasks within a Spark job. Each task is independent of the others. Execution nodes are started simultaneously with the Spark engine and exist throughout the lifecycle of the Spark program. If an execution node fails, the program will not stop running; instead, the tasks on the faulty execution node will be rescheduled to run on other working execution nodes.

[0063] The execution side's main functions include running specific tasks within a Spark job and returning the execution results to the driver side. It uses its own block manager to store user-requested Resilient Distributed Datasets (RDDs) in memory. The RDDs are cached within the execution side process, allowing tasks to fully utilize the cached data to accelerate computation during runtime.

[0064] V. Accumulator

[0065] An accumulator is a node in Spark that records the number of events that occur. Accumulators are mainly used to aggregate variable information from the execution side to the driver side. Variables defined in the driver program will have a new copy in each task on the execution side. After each task updates the values ​​of these copies, they are sent back to the driver side for aggregation.

[0066] VI. Listener

[0067] Spark listeners can be used to track the occurrence of events. After each event occurs, Spark will call a specific method of SparkListener, passing in some metrics or executor information as input parameters, such as execution time, number of records read / written, number of bytes read / written, etc.

[0068] Spark components send events to the LiveListenerBus when performing specific operations. The LiveListenerBus contains various types of listener queues, such as a shared queue that executes user-defined listeners, and an appStatus queue that calls Spark application listeners to listen for events. Each listener queue has its own dispatch thread, which sends events to each listener in the AsyncEventQueue's listener queue, ultimately executing the response methods of these listeners.

[0069] VII. Monitoring Component Grafana

[0070] Grafana is a cross-platform, open-source metrics analysis and visualization tool that allows users to query and visualize collected data, providing timely notifications. Grafana is primarily used for visualization, mixed visualization, annotation, notification alerts, and visualization metric alert rules.

[0071] The application scenarios of this application are described below:

[0072] Currently, Spark's monitoring level reaches the task level, but in reality, a user may define multiple methods within a single task.

[0073] For example, in iterating through partitions using `Foreach Partition` or mapping partitions using `Map Partition`, users might perform various complex operations on the data in each partition. Furthermore, if partition skew exists, the execution time will also differ. If technical personnel need to understand the execution status of methods within these partitions, such as the execution time of the methods,...

[0074] To address the aforementioned issues, this application provides a task monitoring method, apparatus, and device to refine the monitoring levels of the Spark engine and achieve method-level monitoring.

[0075] The task monitoring method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the application scenarios described above are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0076] For the execution side of the distributed computing system Spark, the task monitoring method provided in this application embodiment is as follows: Figure 1 As shown, the method includes:

[0077] Step 101: Monitor each method in each task based on the preset monitoring type, obtain monitoring values, and generate monitoring data based on the monitoring object name, monitoring type name, and monitoring values.

[0078] It should be noted that, similar to how a Stage is divided into at least one Task, a Task can also be divided into at least one Method. For example, functions or operators under a Task can be understood as Methods.

[0079] In addition, the monitoring object name, monitoring type name, and monitoring value are components of the monitoring data. Based on user needs, the components of the monitoring data can be added or removed. For example, the monitoring data may include the monitoring object name and monitoring value, or it may include the task name to which the monitoring object belongs, the monitoring object name, the monitoring type name, and the monitoring value. This application does not limit this.

[0080] Step 102: Concatenate the monitoring data belonging to the same task to obtain the target string.

[0081] Step 103: Send a data packet generated based on the target string to the driver so that the driver can listen to each method in the task of the execution end based on the data packet.

[0082] Taking the execution end based on the time consumption monitoring type and monitoring the monitoring object method 01, and obtaining a monitoring value of 64 milliseconds as an example, the monitoring data generated by the execution end includes method 01, time consumption, and 0046ms. Similarly, the monitoring data of other methods under the same task can be obtained.

[0083] Furthermore, the monitoring data of the same task are concatenated, and the string formed by concatenating multiple methods, namely the target string, is sent to the driver in the form of a data packet so that the driver can listen to the information in the target string.

[0084] Therefore, based on the above technical solution, various methods under a task can be monitored, thereby refining the monitoring level of the Spark engine in the distributed computing system.

[0085] The task monitoring method provided in this application will be further described below with reference to embodiments:

[0086] like Figure 2 As shown in the embodiments of this application, the task monitoring method implemented by the execution end includes:

[0087] Step 201: Monitor each method in each task based on the preset monitoring type, obtain the monitoring value, and concatenate the monitoring object name, monitoring type name and monitoring value into a string to generate monitoring data.

[0088] Taking the execution end based on the time-consuming monitoring type and monitoring the monitored object method 01, obtaining a monitoring value of 64 milliseconds as an example, you can choose "-" as the separator to concatenate method 01, time consumption, and 0046ms to generate monitoring data method 01-time consumption-0046ms. In this way, both the monitoring data and the target string output by concatenating the monitoring data are of string type, which is more convenient for compatibility with most data types.

[0089] Specifically, the value accumulated each time the execution is performed can be defined as a string in the form of "monitoring object name - monitoring type name - monitoring value".

[0090] Step 202: Concatenate the monitoring data belonging to the same task to obtain the target string.

[0091] Specifically, the `add` method of the accumulator on the driver side can be overridden to define a string concatenation function, thereby recording and distinguishing different methods in each task. Furthermore, the `merge` method of the accumulator can also be overridden to define a string concatenation function, thereby recording and distinguishing different tasks in each stage. However, the `add` and `merge` methods use different delimiters: the delimiter for the `add` method is '_', and the delimiter for the `merge` method is ','.

[0092] For example, the accumulator uses the add method to concatenate the monitoring data "Method 01-Time-0046ms" and "Method 02-Time-0048ms" to generate the target string "Method 01-Time-0046ms_Method 02-Time-0048ms". Furthermore, the merge method can be used to concatenate the target strings "Method 01-Time-0046ms_Method 02-Time-0048ms" and "Method 01-Time-0046ms_Method 02-Time-0048ms" belonging to the same stage to obtain the string "Method 01-Time-0046ms_Method 02-Time-0048ms,Method 01-Time-0046ms_Method 02-Time-0048ms".

[0093] It should be noted that the accumulator is registered on the driver side. The driver side sends a copy of the accumulator to at least one execution end. Since the add and merge methods of the accumulator are overridden, the copy of the accumulator on the execution end can execute the overridden add and merge methods.

[0094] Additionally, during the registration process, a user-defined name can be included on the accumulator, making it easier for subsequent listeners to filter events.

[0095] Step 203: Send a data packet generated based on the target string to the driver so that the driver can listen to each method in the task of the execution end based on the data packet.

[0096] In one possible embodiment, the data packet can be a heartbeat packet, for example, the execution end sends a heartbeat packet to the driver end based on a preset period.

[0097] Specifically, the heartbeat generator on the execution side can send heartbeat packets to the driver side at a preset period. In addition to the target string information, the heartbeat packet can also include execution side status information, such as execution side status indicators, etc.

[0098] Unlike the method of using communication data packets to transmit information and using heartbeat packets to confirm the working status of the execution end after establishing a communication connection between the execution end and the driver end, the heartbeat packets in this embodiment can be used for data communication of target strings, thereby reducing the communication pressure between the execution end and the driver end.

[0099] like Figure 2 As shown in the embodiments of this application, the task monitoring method implemented on the driver side includes:

[0100] Step 204: Receive the data packet sent by the execution end.

[0101] In one possible embodiment, the received data packet may be a heartbeat packet.

[0102] Specifically, a heartbeat packet containing the target string information can be received through the heartbeat receiver on the driver side.

[0103] Step 205: Parse the data packet to obtain the target string.

[0104] In one possible implementation, the data packet can be a heartbeat packet, based on which the driver can process the heartbeat packet through the scheduler.

[0105] The scheduler mainly includes two types of interfaces: one is the low-level scheduler interface TaskScheduler, which is mainly used for physically scheduling and executing each specific task; the other is the high-level scheduler interface DAGScheduler, which is mainly used to split tasks into different stages, abstract them into task sets, and then submit them to TaskScheduler for specific processing.

[0106] In this embodiment, the heartbeat receiver processes the heartbeat information through TaskScheduler and DAGScheduler, and finally sends the parsed heartbeat data to the event bus ListenerBus via an event. ListenerBus then delivers ListenerEvent type events to the modules used for listening.

[0107] Step 206: If the monitoring value in the target string is updated, generate indicator data based on the target string corresponding to the updated monitoring value, and send the indicator data to the monitoring system.

[0108] Specifically, a listener can be defined and registered with the event bus ListenerBus during driver initialization. Furthermore, a user-defined name can be included in the listener during registration. By listening to and caching ListenerEvents through the listener, the driver can determine the method name, task name, and stage name corresponding to the target string where the monitored value has been updated. Additionally, the listener can filter based on the registered name of the accumulator, thus only filtering out the user-defined accumulators used for monitoring, which speeds up the monitoring process. Moreover, the listener can convert the target string into key-value pairs or other easily interpretable data formats.

[0109] The listener saves the listening results to the AccumUpdates module, so that the Metrics module can obtain the target string information through the custom Source interface and generate metric data, and propagate it according to the preset metric data propagation method.

[0110] The monitoring system further monitors or displays the indicator data. The monitoring system can be a console or an external visualization monitoring system. For example, the monitoring system can be combined with the monitoring component Grafana to visualize the indicator data.

[0111] In summary, the execution end disclosed in this application can generate monitoring data based on the monitoring object name, monitoring type name, and monitoring value. It can also concatenate various monitoring data belonging to the same task to generate a target string, which is then sent to the driver end in the form of a data packet. Furthermore, the driver end can listen to the monitoring values ​​in the target string, for example, through a listener located on the driver end. If the monitoring value is updated, indicator data is generated based on the target string corresponding to the updated monitoring value, and this indicator data is sent to the monitoring system. This monitoring system can be a console or an externally connected visual monitoring system. The generated indicator data can be provided to the monitoring system for further monitoring or display. Therefore, based on the above technical solution, this application can listen to various methods under a task, thereby refining the monitoring level of the Spark engine in the distributed computing system.

[0112] Based on the same technical concept, this application also provides a task monitoring device. The implementation of this device can refer to the implementation of the task monitoring method described above, and the repeated parts will not be described again.

[0113] The task monitoring device provided in this embodiment is applied to the Spark execution end, and the execution end and the Spark driver are connected for communication, such as... Figure 3 As shown, the device includes a first generation module 301, a splicing module 302, and a sending module 303:

[0114] The first generation module 301 is used to monitor each method in each task based on a preset monitoring type, obtain monitoring values, and generate monitoring data based on the monitoring object name, monitoring type name, and monitoring values.

[0115] The splicing module 302 is used to splice together various monitoring data belonging to the same task to obtain the target string.

[0116] The sending module 303 sends a data packet generated based on the target string to the driver, so that the driver can listen to each method in the task of the execution end based on the data packet.

[0117] In one possible embodiment, the first generation module 301 is specifically used for:

[0118] The monitoring data is generated by concatenating the monitoring object name, monitoring type name, and monitoring value into a string.

[0119] In one possible embodiment, the concatenation module 302 can be a copy of the accumulator, which originates from the accumulator registered on the driver side. This copy can be achieved by overriding the `add` method of the accumulator on the driver side, defining the `add` method as a string concatenation function to record and distinguish different methods in each task. Furthermore, the `merge` method of the accumulator can also be overridden, defining the `merge` method as a string concatenation function to record and distinguish different tasks in each stage. Accordingly, the copy of the accumulator can possess concatenation functionality.

[0120] In one possible embodiment, the data packet sent by the sending module 303 is a heartbeat packet. The sending module 303 can be a heartbeat generator, and the sending module 303 is specifically used for:

[0121] Heartbeat packets are sent to the driver at a preset period.

[0122] Based on the same technical concept, this application also provides a task monitoring device. The implementation of this device can refer to the implementation of the task monitoring method described above, and the repeated parts will not be described again.

[0123] The task monitoring device provided in this application embodiment is applied to the Spark driver, and the driver and Spark execution end are connected for communication, such as... Figure 4 As shown, the device includes a receiving module 401, a parsing module 402, and a second generating module 403.

[0124] The receiving module 401 is used to receive data packets sent by the execution end.

[0125] It should be noted that the data packet is generated by the execution end based on the target string. The target string is generated by the execution end by concatenating various monitoring data belonging to the same task. The monitoring data is generated by the execution end based on the monitoring object name, monitoring type name, and monitoring value. The monitoring value is obtained by the execution end by monitoring each method in each task based on the preset monitoring type.

[0126] Parsing module 402 is used to parse data packets to obtain the target string.

[0127] The second generation module 403 is used to generate indicator data based on the target string corresponding to the updated monitoring value if the monitoring value in the target string is updated, and then send the indicator data to the monitoring system.

[0128] In one possible embodiment, the receiving module 401 may be a heartbeat receiver, which receives heartbeat packets and sends them to the parsing module 402 for parsing and processing.

[0129] In one possible embodiment, the parsing module 402 may be a scheduler, which may include a scheduler interface TaskScheduler and a scheduler interface DAGScheduler.

[0130] In one possible embodiment, the second generation module 403 can be a listener, or a combination of a listener and a measurement module. The listener can be registered to the event bus (ListenerBus) during driver initialization. Furthermore, a user-defined name can be included in the listener during registration. The driver listens to and caches events through the listener, determining the method name, task name, and stage name corresponding to the target string where the monitored value has been updated. Additionally, the listener can filter based on the registered name of the accumulator, thus filtering out only the user-defined portion of the accumulator used for monitoring, accelerating the listening process. Furthermore, the listener can convert the target string into key-value pairs or other easily interpretable data formats.

[0131] The listener saves the listening results to the AccumUpdates module, so that the measurement module can obtain the target string information through the custom source interface and generate indicator data, and propagate it to the final sink according to the preset indicator data propagation method.

[0132] Therefore, the task monitoring device disclosed in this application can concatenate strings using modules with concatenation functions, such as accumulators, to record certain characteristics during method execution in distributed computing. Secondly, the task monitoring device disclosed in this application can continuously track changes in the target string through a second generation module, such as a listener. Finally, the obtained indicator data is continuously output from the data source to the final data collection end to achieve indicator collection and visualization.

[0133] Based on the same technical concept, embodiments of this application also provide a computer device, such as... Figure 5 The device shown includes a processor 501 and a memory 502;

[0134] Processor 501 is used to execute computer-executable programs, such as Figure 1 or Figure 2 The method shown was executed.

[0135] Memory 502 is used to store computer executable programs.

[0136] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program. When the computer-executable program is invoked by a computer device, it causes the computer device to perform actions such as... Figure 1 or Figure 2 The method shown.

[0137] The present application has been described above with reference to block diagrams and / or flowcharts illustrating methods, apparatus (systems), and / or computer program products according to embodiments of the present application. It should be understood that a block of a block diagram and / or flowchart, as well as combinations of blocks of block diagrams and / or flowcharts, can be implemented by computer program instructions. These computer program instructions can be provided to a controller of a general-purpose computer, a special-purpose computer, and / or other programmable data processing means to produce a machine such that the instructions, executable via the computer controller and / or other programmable data processing means, create methods for implementing the functions / actions specified in the blocks of the block diagrams and / or flowcharts.

[0138] Accordingly, this application can also be implemented using hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, this application can take the form of a computer program product on a computer-usable or computer-readable storage medium, having computer-usable or computer-readable program code implemented in the medium for use by or in conjunction with an instruction execution system. In the context of this application, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, transmit, or deliver a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A task monitoring method, characterized in that, An execution terminal for the distributed computing system Spark, wherein the execution terminal and the Spark driver are communicatively connected, the method includes: Each method in each task is monitored based on a preset monitoring type to obtain monitoring values, and monitoring data is generated based on the monitoring object name, monitoring type name, and the monitoring values. The rewritten add method is executed to concatenate the monitoring data belonging to the same task to obtain the target string. The rewritten add method is predefined as a string concatenation function to record and distinguish different methods in each task. A data packet generated based on the target string is sent to the driver, so that the driver can listen to each method in the task of the execution end based on the data packet.

2. The method as described in claim 1, characterized in that, The process of generating monitoring data based on the monitored object name, monitoring type name, and the monitoring value includes: The monitoring data is generated by concatenating the monitoring object name, the monitoring type name, and the monitoring value into a string.

3. The method as described in claim 1 or 2, characterized in that, The data packet is a heartbeat packet; Sending the data packet generated based on the target string to the driver includes: The heartbeat packet is sent to the driver at a preset period.

4. A task monitoring method, characterized in that, The method, applied to the Spark driver and communicating with the Spark execution end, includes: The execution terminal receives data packets sent by the execution terminal; wherein the data packets are generated by the execution terminal based on a target string, the target string is generated by the execution terminal executing a rewritten add method to concatenate various monitoring data belonging to the same task, the rewritten add method is predefined as a string concatenation function to record and distinguish different methods in each task, the monitoring data is generated by the execution terminal based on the monitoring object name, monitoring type name and monitoring value, and the monitoring value is obtained by the execution terminal monitoring each method in each task based on a preset monitoring type; The data packet is parsed to obtain the target string; If the monitoring value in the target string is updated, indicator data is generated based on the target string corresponding to the updated monitoring value, and the indicator data is sent to the monitoring system.

5. The method as described in claim 4, characterized in that, The data packet is a heartbeat packet; The step of parsing the data packet to obtain the target string includes: The heartbeat packet is parsed to obtain the target string.

6. A task monitoring device, characterized in that, An execution terminal for Spark, which is communicatively connected to the Spark driver, the device includes: The first generation module is used to monitor each method in each task based on a preset monitoring type, obtain monitoring values, and generate monitoring data based on the monitoring object name, monitoring type name, and the monitoring values. The concatenation module is used to execute the rewritten add method to concatenate various monitoring data belonging to the same task to obtain the target string. The rewritten add method is predefined as a string concatenation function to record and distinguish different methods in each task. The sending module sends a data packet generated based on the target string to the driver, so that the driver can listen to each method in the task of the execution end based on the data packet.

7. The apparatus as claimed in claim 6, characterized in that, The first generation module is specifically used for: The monitoring data is generated by concatenating the monitoring object name, the monitoring type name, and the monitoring value into a string.

8. A task monitoring device, characterized in that, An apparatus for use with a Spark driver, wherein the driver and the Spark execution end are communicatively connected, the apparatus comprising: A receiving module is used to receive data packets sent by the execution end; wherein, the data packets are generated by the execution end based on a target string, the target string is generated by the execution end executing a rewritten add method to concatenate various monitoring data belonging to the same task, the rewritten add method is predefined as a string concatenation function to record and distinguish different methods in each task, the monitoring data is generated by the execution end based on the monitoring object name, monitoring type name, and monitoring value, and the monitoring value is obtained by the execution end monitoring each method in each task based on a preset monitoring type; The parsing module is used to parse the data packet to obtain the target string; The second generation module is used to generate indicator data based on the target string corresponding to the updated monitoring value if the monitoring value in the target string is updated, and then send the indicator data to the monitoring system.

9. A computer device, characterized in that, The device includes a processor and a memory; The memory is used to store computer-executable programs; The processor is configured to execute the computer-executable program, such that the method as claimed in any one of claims 1 to 3 or claims 4 and 5 is performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program, which, when invoked by a computer device, causes the computer device to perform the method as claimed in any one of claims 1 to 3 or claims 4 and 5.

Images