Spark task runtime driver side or executor side performance diagnosis method

Abstract

The present application relates to the spark task runtime driver end or executor end performance diagnosis method. The method is through obtaining the diagnosis request, wherein the diagnosis request is sent by the user end to the preset multiple driver ends, the diagnosis request is used for requesting performance diagnosis for the target driver end and the target executor end, one driver end corresponds to multiple executor ends, the restful interface is called based on the diagnosis request to generate the diagnosis command, the diagnosis command is configured and distributed to the multiple executor ends corresponding to the current driver end, based on the diagnosis command, whether the current driver end is the target driver end is judged, if yes, the diagnosis is carried out based on the diagnosis command, the diagnosis command is also used for judging whether the executor end is the target executor end, if yes, the diagnosis is carried out based on the diagnosis command, so as to solve the problem that the performance diagnosis of the driver end or the executor end of the spark task is difficult in the runtime.

Description

Technical Field

[0001] This invention relates to the field of big data application technology, specifically to a method for performance diagnosis of the driver or executor side during Spark task runtime. Background Technology

[0002] Apache Spark is a fast and general-purpose computing engine designed for large-scale data processing. It is a distributed big data computing engine, typically used for both offline and real-time computing. Spark tasks exist as JVM processes at runtime. JVM stands for Java Virtual Machine, a specification for computing devices. It is a virtual computer implemented by simulating various computer functions on a real computer.

[0003] Therefore, the traditional method for performance diagnostics on Spark's driver and executor sides is to use diagnostic tools such as jConsole, jmap, jstack (which is the core of Java development and includes the Java runtime environment) or Alibaba's open-source Arthas. However, these diagnostic tools are designed for diagnosing a specific process instance. For a distributed computing engine like Spark, diagnosing it is very difficult. It requires determining the server IP address of the process to be diagnosed, logging into the corresponding server to find the corresponding process ID, and finally using the aforementioned traditional JVM diagnostic tools.

[0004] Spark, as a distributed big data computing engine, exhibits characteristics of distributed computing, which makes analyzing the running instances (driver or executor) on each node quite difficult. Typically, one must first determine the server IP address of the driver or executor to be diagnosed, log into that server, further determine the process ID of the corresponding runtime instance, and finally use JVM diagnostic tools. This traditional diagnostic method is extremely cumbersome when used to diagnose a distributed big data computing engine like Spark. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a server delivery method and apparatus to solve the problem in the prior art that it is difficult to diagnose the performance of Spark tasks on the driver or executor side during runtime for distributed computing engines such as Spark.

[0006] According to a first aspect of the present invention, a performance diagnosis method for the driver or executor side of a Spark task runtime is applied to each of a preset plurality of driver sides, comprising:

[0007] Obtain a diagnostic request; wherein the diagnostic request is sent by the user to multiple preset driver terminals; the driver terminal has a built-in RESTful interface, and the diagnostic request is used to request performance diagnosis for the target driver terminal and the target executor terminal, with one driver terminal corresponding to multiple executor terminals;

[0008] Based on the diagnostic request, the RESTful interface is invoked to generate a diagnostic command;

[0009] The diagnostic command configuration is distributed to multiple executor ends corresponding to the current driver end;

[0010] Based on the diagnostic command, it is determined whether the current driver is the target driver. If so, a diagnosis is performed based on the diagnostic command. The diagnostic command is also used to allow the executor to determine whether it is the target executor. If so, a diagnosis is performed based on the diagnostic command.

[0011] Furthermore, determining whether the current driver is the target driver includes:

[0012] Get the IP address of the server where the current driver is located and the IP address of the server where the target driver is located;

[0013] Determine if the IP address of the server where the current driver is located is the same as the IP address of the server where the target driver is located. If they are the same, determine that the current driver is the target driver.

[0014] Furthermore, the executor determines whether it is the target executor by:

[0015] The executor obtains the IP address and ID of its own server and the IP address and ID of the target executor's server.

[0016] The executor determines whether its own server IP address and ID match the target executor's server IP address and ID. If they match, the executor is identified as the target executor.

[0017] Furthermore, the step of configuring and distributing the diagnostic command to multiple executor ends corresponding to the current driver end includes:

[0018] The diagnostic command configuration is distributed to multiple executors corresponding to the current driver using Spark broadcast.

[0019] Furthermore, including:

[0020] Each of the multiple driver or executor ends has a built-in timer thread;

[0021] The driver is used to poll the timed thread to check whether a diagnostic command has been generated; if it has been generated, the step "distribute the diagnostic command to the multiple executor ends corresponding to the current driver" is executed.

[0022] The executor is used to poll the timed thread to check whether a diagnostic command has been received; if received, the step "distribute the diagnostic command to multiple executor ends corresponding to the current driver" is executed.

[0023] Furthermore, the diagnostic commands include a first instruction for starting the Arthas diagnostic service or a second instruction for stopping the Arthas diagnostic service.

[0024] Furthermore, the diagnostic command includes starting the Arthas diagnostic service, and correspondingly, based on starting the Arthas diagnostic service, the diagnostic command performs the following diagnostics:

[0025] Based on the first instruction, the Arthas diagnostic service is started. The current driver starts the pre-integrated Arthas diagnostic service and registers with the pre-configured Arthas tunnel server service. Online performance diagnosis is performed on the current driver or executor through the tunnel server web page.

[0026] According to a second aspect of the present invention, a performance diagnostic device for the driver or executor side of Spark runtime includes:

[0027] The acquisition module is used to acquire diagnostic requests; wherein the diagnostic requests are sent by the user terminal to multiple preset driver terminals; the driver terminal has a built-in RESTful interface, and the diagnostic requests are used to request performance diagnostics for the target driver terminal and the target executor terminal, with one driver terminal corresponding to multiple executor terminals;

[0028] The generation module is used to generate diagnostic commands by calling the RESTful interface based on the diagnostic request;

[0029] The configuration module is used to distribute the diagnostic command configuration to multiple executor ends corresponding to the current driver end;

[0030] The diagnostic module is used to determine whether the current driver is the target driver based on the diagnostic command, and if so, to perform a diagnosis based on the diagnostic command; the diagnostic command is also used to allow the executor to determine whether it is the target executor, and if so, to perform a diagnosis based on the diagnostic command.

[0031] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0032] This invention allows users to designate a specific driver or executor for diagnosis as the target driver or executor. A diagnostic request for the target driver or executor is sent to each of a pre-defined set of drivers. Each driver generates a diagnostic command based on the request and distributes it to its corresponding executors. The current driver and its executors independently determine if they are the target driver or executor. If so, they invoke the driver's built-in RESTful interface to initiate the diagnostic command for online diagnosis. This approach eliminates the need to pre-determine the IP address of the server hosting the target driver or executor or to log in to the server. It allows for direct online diagnosis of the running Spark driver or executor, improving diagnostic flexibility.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] Figure 1 This is a flowchart illustrating a performance diagnosis method for the driver or executor side of a Spark task runtime, according to an exemplary embodiment.

[0036] Figure 2This is a schematic block diagram of a Spark task runtime driver or executor performance diagnostic device according to an exemplary embodiment.

[0037] Figure 3 This is a schematic diagram illustrating the interaction process of a Spark task runtime driver or executor performance diagnosis method according to an exemplary embodiment.

[0038] In the diagram: 1-Acquisition module; 2-Generation module; 3-Configuration module; 4-Judgment and diagnosis module. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0040] Reference Figure 1 and Figure 3 , Figure 1 This is a flowchart illustrating the performance diagnosis methods for the driver or executor side during Spark task runtime. Figure 3 This is a diagram illustrating the interaction process of performance diagnostic methods on the driver or executor side during Spark task runtime, including:

[0041] Step S11: Obtain a diagnostic request; wherein the diagnostic request is from the user end; the driver end has a built-in RESTful interface, and the diagnostic request is used to request performance diagnostics for the target driver end and the target executor end, with one driver end corresponding to multiple executor ends;

[0042] Step S12: Based on the diagnostic request, call the RESTful interface to generate a diagnostic command;

[0043] Step S13: Distribute the diagnostic command configuration to multiple executor ends corresponding to the current driver end;

[0044] Step S14: Based on the diagnostic command, determine whether the current driver is the target driver. If so, call the RESTful interface to start the diagnostic command for diagnosis. The diagnostic command is also used for the executor to determine whether it is the target executor. If so, call the RESTful interface to start the diagnostic command for diagnosis.

[0045] This invention enables online performance diagnostics for the driver or executor components during Spark task runtime. In practice, based on steps S11 and S12, the target driver or executor is the one being diagnosed by the user. When a Spark task is running, the user, wanting to diagnose a specific driver or executor, sends a diagnostic request to each of a pre-defined set of drivers. One driver corresponds to multiple executors; the driver acts as the control unit, and the executors as the execution units. Additionally, a small RESTful interface service is started on the Spark task's driver using a separate thread. This RESTful interface is used to invoke diagnostic commands, which are generated by the driver based on the user's diagnostic request.

[0046] In step S13, each Spark task has one driver and multiple executors. The driver acts as the leader, and the executors act as employees; the driver manages each executor. For multiple Spark tasks with multiple drivers, each driver distributes diagnostic command configurations to the multiple executors it manages.

[0047] In step S14, each driver or executor can determine whether it matches the target driver or executor. In addition, calling the RESTful interface is to start the diagnostic command. If it is not called, the diagnostic command will not be started. Each driver or executor can determine whether it is the same as the target driver or executor. If they are the same, the RESTful interface is called to start the diagnostic command. If they are different, the diagnostic command will not be started.

[0048] Understandably, this invention allows users to designate a specific driver or executor for diagnosis as the target driver or executor. The user sends a diagnostic request for the target driver or executor to each of a pre-defined set of drivers. The driver generates a diagnostic command based on the request and distributes it to the corresponding executors. The current driver and its executors independently determine if they are the target driver or executor. If so, they invoke the driver's built-in RESTful interface to initiate the diagnostic command for online diagnosis of the current driver or corresponding executor. This approach eliminates the need to pre-determine the IP address of the server hosting the target driver or executor, or to log in to the server. It allows for direct online diagnosis of the running Spark driver or executor, improving diagnostic flexibility.

[0049] Preferably, determining whether the current driver is the target driver includes:

[0050] Get the IP address of the server where the current driver is located and the IP address of the server where the target driver is located;

[0051] Determine if the IP address of the server where the current driver is located is the same as the IP address of the server where the target driver is located. If they are the same, determine that the current driver is the target driver.

[0052] Specifically, each driver can determine whether its own server IP is the same as the IP of the target driver server passed to the user client when calling the RESTful interface. If they are the same, the match is successful.

[0053] Preferably, the executor determines whether it is the target executor by:

[0054] The executor obtains the IP address and ID of its own server and the IP address and ID of the target executor's server.

[0055] The executor determines whether its own server IP address and ID match the target executor's server IP address and ID. If they match, the executor is identified as the target executor.

[0056] Specifically, the executor is a distributed running instance of Spark. Each executor can determine whether its own server IP and executor instance number are the same as the IP and executor instance number of the target executor server passed to the client when calling the RESTful interface. If they are the same, the match is successful.

[0057] Preferably, the step of configuring and distributing the diagnostic command to multiple executor ends corresponding to the current driver end includes:

[0058] The diagnostic command configuration is distributed to multiple executors corresponding to the current driver using Spark broadcast.

[0059] Preferably, each of the multiple driver or executor ends has a built-in timer thread;

[0060] The driver is used to poll the timed thread to check whether a diagnostic command has been generated; if it has been generated, the step "distribute the diagnostic command to the multiple executor ends corresponding to the current driver" is executed.

[0061] The executor is used to poll the timed thread to check whether a diagnostic command has been received; if received, the step "distribute the diagnostic command to multiple executor ends corresponding to the current driver" is executed.

[0062] For example, the timed thread is an independent polling thread that executes every 10 seconds by default. It's worth noting that the threads are executed one by one in a polling manner. The polling continues from the first registered thread until the end of the total number of threads. Once the thread flag is set (meaning the timed polling has occurred), the thread function is assigned a value; the thread flag is then cleared, and the next polling begins. If the current polling is a single run, it is not polled again, and the flag is forcibly set. If multiple executions are required, the thread is canceled when the required number of executions is reached, and the thread count is reduced by one.

[0063] Preferably, the diagnostic commands include a first instruction for starting the Arthas diagnostic service or a second instruction for stopping the Arthas diagnostic service.

[0064] Specifically, the technical solution of this invention can both start and stop performance diagnostics on the Spark driver or executor side at runtime. The execution steps are the same, except that starting the corresponding Arthas diagnostic service is done by launching it as a thread, while stopping it is done by stopping the already running Arthas diagnostic service as a thread.

[0065] Preferably, the diagnostic command includes starting the Arthas diagnostic service, and correspondingly, based on starting the Arthas diagnostic service, the diagnostic command performs the following diagnostics:

[0066] Based on the first instruction, the Arthas diagnostic service is started. The current driver starts the pre-integrated Arthas diagnostic service and registers with the pre-configured Arthas tunnel server service. Online performance diagnosis is performed on the current driver or executor through the tunnel server web page.

[0067] Specifically, the technical solution of the present invention also includes the following beneficial effects:

[0068] Performance diagnostics for a specific driver or executor can be performed directly through a unified tunnel server web interface. The entire process does not require specifying the server or logging into it; all the above operations can be performed solely within the tunnel server web interface.

[0069] Reference Figure 2 , Figure 2 This is a schematic block diagram of a performance diagnostic device for the Spark driver or executor at runtime, such as... Figure 2 As shown, the device includes:

[0070] Module 1 is used to acquire diagnostic requests; wherein the diagnostic requests are sent by the user terminal to multiple preset driver terminals; the driver terminal has a built-in RESTful interface, and the diagnostic requests are used to request performance diagnostics for the target driver terminal and the target executor terminal, with one driver terminal corresponding to multiple executor terminals;

[0071] Generation module 2 is used to generate diagnostic commands by calling the RESTful interface based on the diagnostic request;

[0072] Configuration module 3 is used to distribute the diagnostic command configuration to multiple executor ends corresponding to the current driver end;

[0073] The diagnostic module 4 is used to determine whether the current driver is the target driver based on the diagnostic command, and if so, to perform a diagnosis based on the diagnostic command; the diagnostic command is also used to allow the executor to determine whether it is the target executor, and if so, to perform a diagnosis based on the diagnostic command.

[0074] Understandably, this invention allows users to designate a specific driver or executor for diagnosis as the target driver or executor. The user sends a diagnostic request for the target driver or executor to each of a pre-defined set of drivers. The driver generates a diagnostic command based on the request and distributes it to the corresponding executors. The current driver and its executors independently determine if they are the target driver or executor. If so, they invoke the driver's built-in RESTful interface to initiate the diagnostic command for online diagnosis of the current driver or corresponding executor. This approach eliminates the need to pre-determine the IP address of the server hosting the target driver or executor, or to log in to the server. It allows for direct online diagnosis of the running Spark driver or executor, improving diagnostic flexibility.

[0075] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0076] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0077] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0078] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0079] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0081] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0082] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0083] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for performance diagnosis of the driver or executor side during Spark task runtime, characterized in that, It applies to each of the multiple preset driver endpoints, including: Obtain a diagnostic request; wherein the diagnostic request is sent by the user to multiple preset driver terminals; the driver terminal has a built-in RESTful interface, and the diagnostic request is used to request performance diagnosis for the target driver terminal and the target executor terminal, with one driver terminal corresponding to multiple executor terminals; Based on the diagnostic request, the RESTful interface is invoked to generate a diagnostic command; The diagnostic command configuration is distributed to multiple executor ends corresponding to the current driver end; Based on the diagnostic command, it is determined whether the current driver is the target driver. If so, a diagnosis is performed based on the diagnostic command. The diagnostic command is also used to allow the executor to determine whether it is the target executor. If so, a diagnosis is performed based on the diagnostic command. Determining whether the current driver is the target driver includes: Get the IP address of the server where the current driver is located and the IP address of the server where the target driver is located; Determine if the IP address of the server where the current driver is located is the same as the IP address of the server where the target driver is located. If they are the same, determine that the current driver is the target driver.

2. The method according to claim 1, characterized in that, The executor determines whether it is the target executor by: The executor obtains the IP address and ID of its own server and the IP address and ID of the target executor's server. The executor determines whether its own server IP address and ID match the target executor's server IP address and ID. If they match, the executor is identified as the target executor.

3. The method according to claim 1, characterized in that, The step of configuring and distributing the diagnostic command to multiple executor ends corresponding to the current driver end includes: The diagnostic command configuration is distributed to multiple executors corresponding to the current driver using Spark broadcast.

4. The method according to claim 1, characterized in that, include: Each of the multiple driver or executor ends has a built-in timer thread; The driver is used to poll the timed thread to check whether a diagnostic command has been generated; if it has been generated, the step "distribute the diagnostic command to the multiple executor ends corresponding to the current driver" is executed. The executor is used to poll the timed thread to check whether a diagnostic command has been received; if received, the step "distribute the diagnostic command to multiple executor ends corresponding to the current driver" is executed.

5. The method according to claim 1, characterized in that, The diagnostic commands include a first instruction for starting the Arthas diagnostic service or a second instruction for stopping the Arthas diagnostic service.

6. The method according to claim 5, characterized in that, The diagnostic command includes starting the Arthas diagnostic service. Correspondingly, based on starting the Arthas diagnostic service, the diagnostic command performs the following diagnostics: Based on the first instruction, the Arthas diagnostic service is started. The current driver starts the pre-integrated Arthas diagnostic service and registers with the pre-configured Arthas tunnel server service. Online performance diagnosis is performed on the current driver or executor through the tunnel server web page.

7. A performance diagnostic device for the Spark driver or executor at runtime, characterized in that, include: The acquisition module is used to acquire diagnostic requests; wherein the diagnostic requests are sent by the user terminal to multiple preset driver terminals; the driver terminal has a built-in RESTful interface, and the diagnostic requests are used to request performance diagnostics for the target driver terminal and the target executor terminal, with one driver terminal corresponding to multiple executor terminals; The generation module is used to generate diagnostic commands by calling the RESTful interface based on the diagnostic request; The configuration module is used to distribute the diagnostic command configuration to multiple executor ends corresponding to the current driver end; The diagnostic module is used to determine whether the current driver is the target driver based on the diagnostic command, and if so, to perform a diagnosis based on the diagnostic command; the diagnostic command is also used to allow the executor to determine whether it is the target executor, and if so, to perform a diagnosis based on the diagnostic command. Determining whether the current driver is the target driver includes: Get the IP address of the server where the current driver is located and the IP address of the server where the target driver is located; Determine if the IP address of the server where the current driver is located is the same as the IP address of the server where the target driver is located. If they are the same, determine that the current driver is the target driver.

Images