Method, device and server for terminating tasks submitted by a client to a hadoop cluster

Abstract

The method, device and server for terminating a task submitted by a client to a Hadoop cluster provided by the present disclosure relate to data processing technology and include: obtaining a termination task request, the termination task request including a third session identifier corresponding to a task to be terminated; determining a second task identifier of a task currently being executed according to the termination task request, and determining a second session identifier corresponding to the second task identifier according to the second task identifier; matching the third session identifier with the second session identifier, and if it is determined that the third session identifier matches the second session identifier, terminating the task submitted by the client to the Hadoop cluster corresponding to the second session identifier. This scheme can bind the task submitted by the client to the Hadoop cluster and the client session corresponding to the task through the session identifier, at the session level, to determine the task ownership and find the corresponding relationship between the task and the session, which helps to reduce misoperations and thus saves time and resources.

Description

Technical Field

[0001] This disclosure relates to data processing technology, and more particularly to a method, apparatus, and server for terminating tasks submitted by a client to a Hadoop cluster. Background Technology

[0002] Currently, with the continuous expansion of data scale, most users choose to use big data platforms based on the Hadoop ecosystem to process data in order to solve complex problems such as the storage and analysis of massive amounts of data. The Hadoop ecosystem uses Yet Another Resource Negotiator (YARN) as a resource manager to provide unified resource management and scheduling for upper-layer applications. For tasks submitted by clients to the Hadoop cluster, YARN assigns a unique task identifier (usually an alphanumeric combination). When users need to view the task's running status or terminate a task, they can find the corresponding task identifier to query task running information or end the task.

[0003] In existing technologies, the task identifier of a specific task can be found by using methods such as specifying the job name when submitting the task, assigning a specific queue in YARN, the system time when submitting the task, the system username of the system submitting the task, and the Internet Protocol (IP) address of the client submitting the task, thereby terminating the task.

[0004] However, in the above methods, such as using YARN to allocate a specified queue to help find the task identifier of a specific task, there may be multiple client sessions with the same task identifier for the queue, which is not unique. This may cause the tasks in multiple client sessions to terminate, resulting in a erroneous termination operation. Summary of the Invention

[0005] This disclosure provides a method, apparatus, and server for terminating tasks submitted by a client to a Hadoop cluster, in order to solve the problem of accidental termination caused by terminating tasks in multiple client sessions in the prior art.

[0006] According to a first aspect of this disclosure, a method for terminating a task submitted by a client to a Hadoop cluster is provided, comprising:

[0007] Obtain a task termination request, wherein the task termination request includes: a third session identifier corresponding to the task to be terminated;

[0008] Based on the task termination request, determine the second task identifier of the currently executing task, and based on the second task identifier, determine the second session identifier corresponding to the second task identifier;

[0009] The third session identifier is matched with the second session identifier. If the third session identifier matches the second session identifier, the task submitted by the client corresponding to the second session identifier to the Hadoop cluster is terminated.

[0010] In one implementation, terminating the task submitted to the Hadoop cluster by the client corresponding to the second session identifier includes:

[0011] Based on the first process identifier included in the second session identifier, a process termination instruction is sent to the client to terminate the process in the client corresponding to the second session identifier;

[0012] Terminate the task currently being executed corresponding to the second session identifier.

[0013] In one possible implementation, sending a process termination instruction to the client based on a first process identifier included in the second session identifier to terminate the process in the client corresponding to the second session identifier includes:

[0014] Based on the first process identifier included in the second session identifier, determine all child processes under the process corresponding to the first process identifier;

[0015] A process termination command is sent to the client to terminate all child processes under the process corresponding to the first process identifier.

[0016] One possible implementation also includes:

[0017] The system receives a task request submitted by a client and assigns a first task identifier to the first task in the task request based on the task request. The task request includes a first task and a first session identifier corresponding to the first task. The first session identifier is used to characterize the current session information of the client that submitted the task request.

[0018] In one possible implementation, the first session identifier includes one or more combinations of the following:

[0019] The first process identifier corresponding to the current session of the client, the Internet Protocol address corresponding to the client, the username of the user who logged into the client, and the initial login time of the user who logged into the client.

[0020] In one possible implementation, matching the third session identifier with the second session identifier includes:

[0021] The third session identifier, which includes the first process identifier, Internet Protocol address, username, and initial login time, is matched one by one with the second session identifier, which includes the first process identifier, Internet Protocol address, username, and initial login time. If all matches are successful, it is determined that the third session identifier matches the second session identifier.

[0022] According to a second aspect of this disclosure, an apparatus is provided for terminating a task submitted by a client to a Hadoop cluster, comprising:

[0023] The acquisition unit is used to acquire a task termination request, wherein the task termination request includes: a third session identifier corresponding to the task to be terminated;

[0024] The determining unit is configured to determine, based on the task termination request, a second task identifier of the currently executing task, and a second session identifier corresponding to the second task identifier;

[0025] The termination unit is used to match the third session identifier with the second session identifier. If it is determined that the third session identifier matches the second session identifier, the task submitted by the client corresponding to the second session identifier to the Hadoop cluster is terminated.

[0026] According to a third aspect of this disclosure, a server is provided, including a memory and a processor; wherein,

[0027] The memory is used to store computer programs;

[0028] The processor is configured to read a computer program stored in the memory and execute, according to the computer program in the memory, the method for terminating a client-submitted task to the Hadoop cluster as described in the first aspect.

[0029] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, a method for terminating a task submitted by a client to a Hadoop cluster as described in the first aspect is implemented.

[0030] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method for terminating tasks submitted by a client to a Hadoop cluster as described in the first aspect.

[0031] The method, apparatus, and server disclosed herein for terminating tasks submitted by a client to a Hadoop cluster include: obtaining a task termination request, the task termination request including a third session identifier corresponding to the task to be terminated; determining a second task identifier of the currently executing task based on the task termination request, and determining a second session identifier corresponding to the second task identifier based on the second task identifier; matching the third session identifier with the second session identifier, and if the third session identifier matches the second session identifier, terminating the task submitted by the client to the Hadoop cluster corresponding to the second session identifier. The method, apparatus, and server provided in this solution for terminating tasks submitted by a client to a Hadoop cluster can bind the task submitted by the client to the Hadoop cluster and the corresponding client session to the task with session-level granularity through the session identifier, thereby determining task ownership and identifying the correspondence between tasks and sessions, which helps reduce erroneous operations and saves time and resources. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating a method for terminating a task submitted by a client to a Hadoop cluster, as shown in an exemplary embodiment of this disclosure.

[0034] Figure 2 A flowchart illustrating a method for terminating a task submitted by a client to a Hadoop cluster, as shown in another exemplary embodiment of this disclosure;

[0035] Figure 3 This is a schematic diagram illustrating a process for terminating a task submitted by a client to a Hadoop cluster, as shown in another exemplary embodiment of this disclosure.

[0036] Figure 4 A structural diagram of an apparatus for terminating a client-submitted task to a Hadoop cluster, as illustrated in an exemplary embodiment of this disclosure;

[0037] Figure 5 This is a structural diagram of a server shown as an exemplary embodiment of the present disclosure. Detailed Implementation

[0038] Currently, with the continuous expansion of data scale, most users choose to use big data platforms based on the Hadoop ecosystem to process data in order to solve complex problems such as the storage and analysis of massive amounts of data. In enterprise applications, increasingly complex application scenarios are also emerging. When multiple user business applications are deployed on the same Hadoop cluster, cluster resources can easily become chaotic and disordered during use. Typically, system administrators allocate corresponding cluster resources for different business scenarios. If the resource allocation method is unreasonable, or if the task concurrency is high, overall cluster resources often become strained, leading to task backlog. In this case, it is necessary to appropriately terminate or postpone unreasonable or unimportant tasks based on resource usage to ensure that relatively important tasks are executed in a timely manner. The Hadoop ecosystem uses YARN as a resource manager to provide unified resource management and scheduling for upper-layer applications. For tasks submitted to the Hadoop cluster by clients, YARN assigns a unique task identifier (usually an alphanumeric combination). When users need to view the task's running status or terminate a task, they can find the corresponding task identifier to query task running information or end the task. In real-world applications, authorized personnel within an organization can simultaneously operate the cluster, read and write files, and submit programs, resulting in a large number of tasks running concurrently. Therefore, locating the target task among numerous running tasks, identifying the actual submitter of that task, and subsequently communicating the task's progress becomes crucial for managing cluster tasks.

[0039] There are currently five common ways to identify the task identifier when submitting a task:

[0040] 1) Filter by name in YARN Explorer by specifying the job name when submitting the task; this method is the most common and accurate.

[0041] 2) Assign a specified queue through YARN, and filter by queue in the YARN Resource Manager;

[0042] 3) Confirm the submission using the system time at the time of task submission via timestamp in YARN Explorer;

[0043] 4) The system username used to submit tasks is typically different for different users;

[0044] 5) Match the specific task configuration information in YARN Resource Manager using the IP address of the client that submitted the task;

[0045] However, while specifying task names according to a unified standard can help locate task identifiers, inexperienced personnel may easily overlook this step. If this step is skipped, even the default task name in the Hadoop cluster may not be sufficient to determine the correct task affiliation. Explicitly specifying a fixed task name in the program lacks flexibility, making dynamic assignment impossible when there is high task concurrency. Furthermore, it is often ineffective for third-party programs to specify task names. Additionally, when multiple programs use the same task name, it becomes impossible to distinguish the actual submitter of each task, making it even more difficult for administrators to determine the specific task affiliation and thus impossible to identify the specific task identifier.

[0046] Furthermore, besides specifying the job name when submitting a task, methods such as assigning a specific queue via YARN, using the system time at the time of task submission, using the system username at the time of task submission, and using the IP address of the client submitting the task can only narrow down the target range and cannot accurately locate the task identifier belonging to a client session. For example, using a specific queue assigned by YARN to help find the task identifier of a specific task may result in the queue corresponding to task identifiers of multiple client sessions, which is not unique. This could lead to the termination of tasks in multiple client sessions, resulting in erroneous termination operations.

[0047] In summary, the five methods for determining task identifiers commonly used in most scenarios can only narrow down the target range when the Hadoop cluster environment is complex, with a large number of technical personnel using the cluster and the tasks likely originating from different Linux-based clients. System administrators, business managers, and even the staff submitting the specific tasks may not be able to accurately identify the task identifier and corresponding client session for a particular task, leading to the termination of tasks in multiple client sessions and resulting in accidental termination.

[0048] To address the aforementioned technical issues, the solution provided in this disclosure allows for the binding of tasks submitted by clients to the Hadoop cluster with the corresponding client sessions at a session-level precision using session identifiers. This helps determine task ownership, identify the correspondence between tasks and sessions, reduce erroneous operations, and more accurately identify specific tasks and their corresponding clients, thereby terminating abnormal programs and saving time and resources.

[0049] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0050] Figure 1 This is a flowchart illustrating a method for terminating a task submitted by a client to a Hadoop cluster, as an exemplary embodiment of this disclosure.

[0051] like Figure 1 As shown in this embodiment, the method for terminating a client-submitted task to the Hadoop cluster includes:

[0052] Step 101: Obtain the task termination request. The task termination request includes: the third session identifier corresponding to the task to be terminated.

[0053] The method provided in this disclosure can be executed by a server with computing capabilities, such as a computer or other similar device.

[0054] This server can be based on the Linux operating system and can host a Hadoop cluster. This server can receive requests to terminate tasks.

[0055] The task termination request may include a third session identifier corresponding to the task to be terminated. This session identifier is used to represent the client's session information.

[0056] Specifically, the third session identifier corresponding to the task that needs to be terminated can be obtained through multiple channels.

[0057] For example, it can be obtained through business means. Specifically, by utilizing the correspondence between session identifiers recorded in the business system and specific business operations, the business side can proactively propose session identifiers that need to be terminated as needed; or the business side can identify session identifiers that have exceeded the time limit (exceeding the time limit means that the task execution time in the session is too long, such as 24 or 48 hours without ending), thereby determining the third session identifier.

[0058] It can also be obtained through technical means. Specifically, one can find obviously abnormal processes in the client's load (such as CPU load, memory usage, etc.), find the process identifier of the parent process corresponding to the process through the process tree, and combine it with the username of the user who logged into the client and the IP address of the client to determine the third session identifier.

[0059] Step 102: Based on the task termination request, determine the second task identifier of the currently executing task, and based on the second task identifier, determine the second session identifier corresponding to the second task identifier.

[0060] Specifically, based on the task termination request, the second task identifier of the currently executing task can be obtained through the YARN Resource Manager. Then, the second session identifier corresponding to this second task identifier can be queried.

[0061] Step 103: Match the third session identifier with the second session identifier. If the third session identifier matches the second session identifier, terminate the task submitted by the client corresponding to the second session identifier to the Hadoop cluster.

[0062] Specifically, the third session identifier can be matched with the second session identifier. If the corresponding content included in the third session identifier and the second session identifier is the same, then the third session identifier is determined to match the second session identifier; otherwise, the third session identifier is determined to not match the second session identifier.

[0063] Specifically, if the third session identifier is determined to match the second session identifier, then based on the second session identifier, at least one currently executing task submitted to the Hadoop cluster by a client corresponding to the second session identifier can be queried and these tasks can be terminated.

[0064] The method for terminating a client-submitted task to a Hadoop cluster, provided in this disclosure, includes: obtaining a task termination request, the task termination request including a third session identifier corresponding to the task to be terminated; determining a second task identifier for the currently executing task based on the task termination request, and determining a second session identifier corresponding to the second task identifier based on the second task identifier; matching the third session identifier with the second session identifier, and if the third session identifier matches the second session identifier, terminating the task submitted by the client to the Hadoop cluster corresponding to the second session identifier. In this method, the task submitted by the client to the Hadoop cluster and the corresponding client session can be bound at a session-level fine-grained manner using the session identifier to determine task ownership and identify the correspondence between tasks and sessions. This helps reduce erroneous operations and allows for more accurate identification of a specific task and its corresponding client, thereby terminating abnormal programs and saving time and resources.

[0065] Figure 2 This is a flowchart illustrating a method for terminating a task submitted by a client to a Hadoop cluster, as shown in another exemplary embodiment of this disclosure.

[0066] like Figure 2 As shown in this embodiment, the method for terminating a client-submitted task to the Hadoop cluster includes:

[0067] Step 201: Obtain the task termination request. The task termination request includes: the third session identifier corresponding to the task to be terminated.

[0068] Specifically, a task termination request can be obtained. This task termination request includes the third-party session identifier corresponding to the task to be terminated.

[0069] In one possible implementation, the method further includes: receiving a task request submitted by a client, and assigning a first task identifier to the first task in the task request according to the task request; the task request includes the first task and a first session identifier corresponding to the first task; the first session identifier is used to characterize the current session information of the client that submitted the task request.

[0070] Specifically, before obtaining the task termination request, the process may further include: receiving the task request submitted by the client using the YARN resource manager in the Hadoop cluster, and assigning a first task identifier to the first task in the task request based on the task request. This first task identifier can be used to uniquely identify its corresponding first task.

[0071] The client can be based on the Linux operating system.

[0072] Specifically, a task request submitted by a client to the Hadoop cluster on the server may include a first task and a first session identifier corresponding to that first task. The first session identifier includes the current session information of the client that submitted the task request.

[0073] Specifically, each task corresponds to a unique task identifier, and each task corresponds to a unique session identifier. However, each session identifier can correspond to multiple tasks, and thus each session identifier can correspond to multiple task identifiers.

[0074] In one possible implementation, the first session identifier includes one or more combinations of the following:

[0075] The first process identifier corresponding to the current client session, the Internet Protocol address corresponding to the client, the username of the user who logged into the client, and the initial login time of the user.

[0076] Specifically, the first session identifier, which includes the current session information of the client, may include one or more combinations of the following: the first process identifier corresponding to the current session of the client, the Internet Protocol address corresponding to the client, the username of the user who logged into the client, and the initial login time of the user who logged into the client.

[0077] The first process identifier corresponds to the first process assigned to the client's current session. This first process identifier will not change during the duration of this session.

[0078] The username of the user logging into the client can refer to either a test account or a formal account used to log into the client.

[0079] The initial login time of the login client can refer to the initial login timestamp information of the login client.

[0080] Step 202: Based on the task termination request, determine the second task identifier of the currently executing task, and based on the second task identifier, determine the second session identifier corresponding to the second task identifier.

[0081] Specifically, the principle and implementation of step 202 are similar to those of step 102, and will not be repeated here.

[0082] Step 203: Match the first process identifier, Internet Protocol address, username, and initial login time included in the third session identifier with the first process identifier, Internet Protocol address, username, and initial login time included in the second session identifier. If all matches are successful, it is determined that the third session identifier matches the second session identifier.

[0083] Specifically, the first process identifier corresponding to the client session, the client's Internet Protocol address, the username of the user who logged into the client, and the initial login time of the user who logged into the client, which are included in the third session identifier, can be matched one by one with the corresponding content in the second session identifier. If all matches are successful, it is determined that the third session identifier matches the second session identifier.

[0084] Furthermore, multiple third-session identifiers corresponding to multiple tasks that need to be terminated can be obtained. Multiple second-task identifiers for multiple currently executing tasks can also be obtained. These multiple third-session identifiers can be compared one-to-one with the multiple second-task identifiers.

[0085] Step 204: If it is determined that the third session identifier matches the second session identifier, then according to the first process identifier included in the second session identifier, a process termination instruction is sent to the client to terminate the process in the client corresponding to the second session identifier.

[0086] Specifically, if it is determined that the third session identifier matches the second session identifier, a process termination instruction can be generated based on the first process identifier included in the second session identifier. The process termination instruction can then be sent to the client based on the client's Internet Protocol address and the username of the user who logged into the client, as included in the second session identifier, to terminate the corresponding process in the client corresponding to the second session identifier.

[0087] In one possible implementation, all child processes under the process corresponding to the first process identifier are determined based on the first process identifier included in the second session identifier.

[0088] Specifically, based on the first process identifier included in the second session identifier, the first process corresponding to the first process identifier can be determined, and the process tree can be queried based on the first process to determine all child processes with the first process as the parent process. Thus, all child processes under the first process corresponding to the first process identifier can be obtained.

[0089] Then, a process termination command is sent to the client to terminate all child processes under the process corresponding to the first process identifier.

[0090] Based on all child processes under the process corresponding to the first process identifier, a process termination instruction can be generated and sent to the client to terminate all child processes under the process corresponding to the first process identifier.

[0091] Furthermore, this can be achieved using the interactive expect command based on Tcl scripts:

[0092] The client is located based on the Internet Protocol address of the client included in the second session identifier and the username of the user who logged in to the client; and the pre-set configuration table is queried based on the username of the user who logged in to the client included in the second session identifier to obtain the password for logging into the client. The pre-set configuration table includes the mapping relationship between the username of the user who logged in to the client and the password for logging into the client.

[0093] Then, based on the obtained login password, log in to the client and terminate all child processes under the process corresponding to the first process identifier included in the second session identifier within the client.

[0094] Step 205: Terminate the task being executed corresponding to the second session identifier.

[0095] Specifically, after sending the process termination command to the client, the system can query the currently executing task corresponding to the second session identifier based on the second session identifier, and terminate the currently executing task corresponding to the second session identifier.

[0096] The YARN resource manager receives task requests from clients and operates in two modes: client and cluster. Existing technologies often simply stop the process on the Linux-based client without promptly terminating the YARN task. This can easily lead to the YARN resource manager not releasing resources. The method provided in this embodiment avoids the inconsistency in real-world Hadoop production environments where the client process has been terminated, but the YARN resource manager has not yet finished the task. It promptly releases unnecessary resources, improving resource utilization efficiency.

[0097] Figure 3 This is a schematic diagram illustrating the process of terminating a task submitted by a client to a Hadoop cluster, as an exemplary embodiment of this disclosure.

[0098] Specifically, a Hadoop cluster can be hosted on a Linux operating system server, and the client operating system can also be Linux. Linux supports multiple users and multiple tasks. When different users log in to the same client to submit tasks to the Hadoop cluster, the Linux operating system can create a new session (shell session) for each client user to enable interaction between the user and the system. When a user logs in to the client, the session already exists and has the environment to interact with the Hadoop cluster. By identifying the correspondence between YARN tasks and Linux operating system sessions, all tasks submitted to the Hadoop cluster within the lifecycle of a single session can be identified.

[0099] For each session, the Linux operating system on the client assigns a process identifier, which remains unchanged during the session. A globally unique session identifier can be generated based on the current session's process identifier, the client's IP address, the username of the user logged into the client, and the user's initial login time. This session identifier corresponds to a unique client Linux operating system session.

[0100] like Figure 3 As shown, a session identifier can be injected into the current session. Specifically, after the Linux session on the client starts, the session identifier can be appended to the initial client configuration file YARN-site.xml, declared in the XML file in a "key-value" manner, becoming part of the YARN properties in the Hadoop cluster, forming a new, time-sensitive (effective in the current session) Hadoop configuration, which is then referenced by the client. With the session identifier, the client can submit tasks to the Hadoop cluster normally without special operations; when the client sends a task request to the Hadoop cluster, it will also submit the effective client configuration to the cluster. Since the session identifier has been declared and saved, the current session can exit after submitting the required tasks without waiting for the tasks to complete. When exiting the session, the temporary Hadoop configuration is cleared, the client's Hadoop configuration becomes invalid, and it reverts to the initial client configuration state. Furthermore, a session identifier table can be generated based on the session identifier and stored in a database for easy viewing and use later.

[0101] In YARN, all tasks successfully submitted to the cluster will appear in the scheduling page of the resource manager. For tasks submitted by clients, a reasonable interval can be set for cyclic scanning based on the timeliness requirements of monitoring and the load of the Hadoop cluster nodes. Different operations can be performed by scanning the task running status: for example, for tasks in the running state, the task identifier is recorded and retained for comparison when the task is scanned again; for tasks in the completed or failed state, they are directly filtered without recording or retaining information.

[0102] The YARN resource manager identifies task types. For common MapReduce and Spark computation tasks, the Hadoop cluster employs different task management methods, and the paths for storing configuration files also differ between the two types. For tasks in a monitored running state, the task type is determined. For common Spark tasks, the configuration file is located in the current user's HDFS path `.sparkStaging / ApplicationID / _spark_conf_.zip`; for MapReduce tasks, the configuration file is located in the current user's HDFS path `.staging / ApplicationID / job.xml`. Based on the task type of the monitored running task, the system navigates to the path where the configuration file is stored, retrieves the client configuration file for that task, and then obtains the session identifier corresponding to that task.

[0103] Specifically, the client configuration file stores all configuration information. By identifying key-value pairs, the system finds the value corresponding to the session identifier contained in the task and compares it with the session identifier corresponding to the task that needs to be terminated. If a match is found, the YARN task submitted by the corresponding session can be found based on the matched session identifier; if no match is found, the task status needs to be monitored.

[0104] A successfully matched session identifier can be reconstructed using mapping encoding to obtain the process identifier of its parent session. Then, by examining the process tree of the process corresponding to that process identifier, the process identifiers of all child processes whose parent process is that process are identified, arranged sequentially according to their call relationships, with the parent process listed first and the child processes listed last. Once a session identifier is successfully matched, the task identifier for that task is obtained through the YARN Resource Manager.

[0105] In this context, the process belongs to the client's Linux operating system, while the YARN task belongs to the Hadoop cluster. Traditional methods require logging into the specific client and terminating the corresponding process.

[0106] This method uses the `expect` interactive command based on a Tcl script. After configuring a password, it enables self-service remote control of the client, sending commands to the client to terminate the corresponding processes. Specifically, based on the process IDs of all found child processes, they are arranged in order. After terminating all child processes by sending commands, the task corresponding to the corresponding task ID is then terminated.

[0107] Figure 4 This is a structural diagram illustrating an apparatus for terminating a task submitted by a client to a Hadoop cluster, as shown in an exemplary embodiment of this disclosure.

[0108] like Figure 4 As shown, the apparatus 400 provided in this disclosure for terminating tasks submitted by a client to a Hadoop cluster includes:

[0109] Acquisition unit 410 is used to acquire a task termination request, which includes: a third session identifier corresponding to the task to be terminated;

[0110] The determining unit 420 is used to determine the second task identifier of the currently executing task based on the task termination request, and to determine the second session identifier corresponding to the second task identifier based on the second task identifier.

[0111] Termination unit 430 is used to match the third session identifier with the second session identifier. If it is determined that the third session identifier matches the second session identifier, the task submitted by the client corresponding to the second session identifier to the Hadoop cluster is terminated.

[0112] The termination unit 430 is specifically used to send a process termination instruction to the client based on the first process identifier corresponding to the current session of the client, which is included in the second session identifier, so as to terminate the process in the client corresponding to the second session identifier.

[0113] Terminate the task currently being executed corresponding to the second session identifier.

[0114] The termination unit 430 is specifically used to determine all child processes under the process corresponding to the first process identifier based on the first process identifier included in the second session identifier;

[0115] Send a process termination command to the client to terminate all child processes under the process corresponding to the first process identifier.

[0116] The apparatus 400 provided in this disclosure for terminating tasks submitted by a client to a Hadoop cluster further includes:

[0117] The receiving unit 440 is used to receive a task request submitted by a client, and to assign a first task identifier to the first task in the task request according to the task request; the task request includes the first task and the first session identifier corresponding to the first task; the first session identifier is used to represent the current session information of the client that submitted the task request.

[0118] In one possible implementation, the first session identifier includes one or more combinations of the following:

[0119] The first process identifier corresponding to the current client session, the Internet Protocol address corresponding to the client, the username of the user who logged into the client, and the initial login time of the user.

[0120] The termination unit 430 is specifically used to match the first process identifier, Internet Protocol address, username, and initial login time included in the third session identifier with the first process identifier, Internet Protocol address, username, and initial login time included in the second session identifier. If all matches are successful, it is determined that the third session identifier matches the second session identifier.

[0121] Figure 5 This is a structural diagram of a server shown as an exemplary embodiment of the present disclosure.

[0122] like Figure 5 As shown, the server provided in this embodiment includes:

[0123] Memory 501;

[0124] Processor 502; and

[0125] Computer programs;

[0126] The computer program is stored in memory 501 and configured to be executed by processor 502 to implement any of the methods described above for terminating tasks submitted by the client to the Hadoop cluster.

[0127] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement any of the methods described above for terminating a task submitted by a client to a Hadoop cluster.

[0128] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above for terminating tasks submitted by a client to a Hadoop cluster.

[0129] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for terminating a task submitted by a client to a Hadoop cluster, characterized in that, include: Obtain a task termination request, wherein the task termination request includes: a third session identifier corresponding to the task to be terminated; Based on the task termination request, determine the second task identifier of the currently executing task, and based on the second task identifier, determine the second session identifier corresponding to the second task identifier; The third session identifier is matched with the second session identifier. If the third session identifier matches the second session identifier, the task submitted by the client corresponding to the second session identifier to the Hadoop cluster is terminated.

2. The method according to claim 1, characterized in that, The task of terminating the client corresponding to the second session identifier submitted to the Hadoop cluster includes: Based on the first process identifier included in the second session identifier, a process termination instruction is sent to the client to terminate the process in the client corresponding to the second session identifier; Terminate the task currently being executed corresponding to the second session identifier.

3. The method according to claim 2, characterized in that, The step of sending a process termination instruction to the client based on the first process identifier included in the second session identifier to terminate the process in the client corresponding to the second session identifier includes: Based on the first process identifier included in the second session identifier, determine all child processes under the process corresponding to the first process identifier; A process termination command is sent to the client to terminate all child processes under the process corresponding to the first process identifier.

4. The method according to claim 1, characterized in that, Also includes: The system receives a task request submitted by a client and assigns a first task identifier to the first task in the task request based on the task request. The task request includes a first task and a first session identifier corresponding to the first task. The first session identifier is used to characterize the current session information of the client that submitted the task request.

5. The method according to claim 4, characterized in that, The first session identifier includes one or more combinations of the following: The first process identifier corresponding to the current session of the client, the Internet Protocol address corresponding to the client, the username of the user who logged into the client, and the initial login time of the user who logged into the client.

6. The method according to claim 5, characterized in that, The step of matching the third session identifier with the second session identifier includes: The third session identifier, which includes the first process identifier, Internet Protocol address, username, and initial login time, is matched one by one with the second session identifier, which includes the first process identifier, Internet Protocol address, username, and initial login time. If all matches are successful, it is determined that the third session identifier matches the second session identifier.

7. An apparatus for terminating a task submitted by a client to a Hadoop cluster, characterized in that, include: The acquisition unit is used to acquire a task termination request, wherein the task termination request includes: a third session identifier corresponding to the task to be terminated; The determining unit is configured to determine, based on the task termination request, a second task identifier of the currently executing task, and a second session identifier corresponding to the second task identifier; The termination unit is used to match the third session identifier with the second session identifier. If it is determined that the third session identifier matches the second session identifier, the task submitted by the client corresponding to the second session identifier to the Hadoop cluster is terminated.

8. A server, characterized in that, Includes memory and processor; among which, The memory is used to store computer programs; The processor is configured to read a computer program stored in the memory and execute the method described in any one of claims 1-6 according to the computer program in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images