A task processing method, device and storage medium

Abstract

The embodiments of the present application provide a method, device and storage medium for task processing, which are used to reduce the latency of task processing. In the present application, the driver node performs pre-startup according to the startup instruction of the Yarn scheduler, wherein the driver node is selected by the Yarn scheduler from the Spark cluster according to the virtual task request sent by the client; the driver node obtains the target task from the message queue of the executable task and executes the obtained target task, wherein the executable task in the message queue is sent by the client; the driver node sends the execution result of the target task to the message queue, so that the client obtains the execution result of the target task from the message queue. The driver node is pre-selected from the Spark cluster by the Yarn scheduler according to the virtual task request sent by the client, so when there is a target task, the driver node executes directly, saving the time of the Yarn scheduler selecting the driver node and reducing the latency of task processing.

Description

Background Art

[0002] When the client has actual task requirements, the client starts the actual task and sends a request to the Yarn scheduler to execute the actual task. After receiving the actual task request, the Yarn scheduler selects an idle computing node from the Spark cluster managed by the Yarn scheduler to execute the actual task. If there are no idle computing nodes at this time, the actual task needs to wait until there are idle computing nodes before it can execute the actual task. This results in untimely processing of the actual task and increases the processing time of the actual task. Summary of the invention

[0003] The present application provides a task processing method, device and storage medium to reduce the delay of task processing.

[0004] In a first aspect, an embodiment of the present application provides a task processing method, which is applied to a server including a Spark on Yarn architecture, and the method includes:

[0005] The driver node performs pre-startup according to the startup instruction of the Yarn scheduler, where the driver node is selected from the Spark cluster by the Yarn scheduler according to the virtual task request sent by the client;

[0006] The driver node obtains the target task from the message queue of executable tasks and executes the obtained target task, wherein the executable task in the message queue is sent by the client;

[0007] The driver node sends the execution result of the target task to the message queue, so that the client can obtain the execution result of the target task from the message queue.

[0008] In a second aspect, an embodiment of the present application provides a method for task processing, which is applied in a client, and the method includes:

[0009] The client sends the executable task as the target task to the message queue, so that the driver node obtains the target task from the message queue;

[0010] The client detects the message queue and receives the execution results of the target task fed back by the driver node.

[0011] In a third aspect, an embodiment of the present application provides a task processing device, the device comprising: a pre-starting unit, an execution unit and a first sending unit; wherein:

[0012] A pre-start unit, used to perform pre-start according to a start instruction of a Yarn scheduler, wherein a driver node is selected from a Spark cluster by the Yarn scheduler according to a virtual task request sent by a client;

[0013] An execution unit, used to obtain a target task from a message queue of executable tasks and execute the obtained target task, wherein the executable task in the message queue is sent by the client;

[0014] The first sending unit is used to send the execution result of the target task to the message queue, so that the client obtains the execution result of the target task from the message queue.

[0015] In a possible implementation, the execution unit is further configured to: detect a message queue of executable tasks in real time; and when there is an executable task in the message queue of executable tasks, obtain the executable task as a target task.

[0016] In a possible implementation, the execution unit is further used to: analyze the target task; when the data type of the target task is an RDD (Resilient Distributed Dataset) data type, distribute the target task to the execution node, so that the driver node and the execution node jointly execute the target task;

[0017] The execution node is obtained by sending a resource request to the Yarn scheduler after determining that the data type of the target task is the RDD data type.

[0018] In a fourth aspect, an embodiment of the present application provides a task processing device, the device comprising: a second sending unit and a receiving unit, wherein:

[0019] A second sending unit is used to send the executable task as a target task to the message queue, so that the driver node obtains the target task from the message queue;

[0020] The receiving unit is used to detect the message queue and receive the execution result of the target task fed back by the driving node.

[0021] In a possible implementation manner, the second sending unit is further configured to: perform serialization processing on the executable tasks.

[0022] In a possible implementation manner, the second sending unit is further used to: send a virtual task request to the Yarn scheduler, so that the Yarn scheduler selects a driver node for detecting the message queue from the Spark cluster.

[0023] In a fifth aspect, an embodiment of the present application provides a task processing device, comprising: a memory and a processor, wherein the memory is used to store computer instructions; and the processor is used to execute computer instructions to implement the task processing method provided by the embodiment of the present application.

[0024] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the task processing method provided by the embodiment of the present application is implemented.

[0025] The beneficial effects of this application are as follows:

[0026] The present application provides a method, device and storage medium for task processing, wherein the driver node is selected from the Spark cluster by the Yarn scheduler according to the virtual task request sent by the client, and performs pre-start, and detects the message queue of the executable task in real time. When there is an executable task sent by the client in the message queue, the executable task is read from the message queue as the target task, and the target task is executed, the execution result of the target task is determined, and the execution result of the target task is sent to the message queue, so that the client obtains the execution result of the target task. When an executable task arrives, the driver node directly reads the executable task from the message queue, executes it as the target task, determines the execution result, and skips the scheduling time of the Yarn scheduler. By saving the time of the Yarn scheduler selecting the driver node, time consumption is optimized and the delay of task processing is reduced.

[0027] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of the architecture for task processing in related technologies;

[0030] Figure 2 Schematic diagram of application scenarios for task processing provided by this application;

[0031] Figure 3 A system architecture diagram for task processing provided for this application;

[0032] Figure 4 A schematic diagram of sending a virtual task request to obtain a driver node provided by the present application;

[0033] Figure 5A schematic diagram of sending an executable task provided by this application;

[0034] Figure 6 A schematic diagram of a driver node feedback execution result provided in this application;

[0035] Figure 7 A schematic diagram of the overall process of task processing provided for this application;

[0036] Figure 8 A rendering of a task processing provided for this application;

[0037] Fig. 9 A flowchart of a task processing method provided for this application;

[0038] Fig.10 A flowchart of another task processing method provided by this application;

[0039] Fig.11 A structural diagram of a task processing device provided for this application;

[0040] Fig.12 A structural diagram of another task processing device provided by this application;

[0041] Fig.13 A structural diagram of a computing device provided for this application. DETAILED DESCRIPTION

[0042] The architecture and task scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new task scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0043] The following is an explanation of some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art:

[0044] Compute node: A physical machine can be split into multiple nodes, each of which is also called a compute node.

[0045] The driver node, also known as the Spark Driver, is selected from the computing nodes in the Spark cluster when starting a task through the Spark computing framework. It is used to perform global work, such as resource application and task allocation.

[0046] The execution node, also known as Spark Executor, is the node that actually processes data in the task and is responsible for running the task.

[0047] The Yarn scheduler is responsible for resource scheduling and is mainly used to assign computing nodes to tasks.

[0048] Serialization is the process of converting the state information of an executable target task into a form that can be stored or transmitted.

[0049] The following is a brief introduction to the design concept of the embodiments of the present application.

[0050] like Figure 1 As shown, it is a schematic diagram of the architecture of task processing in the related technology. Figure 1 It can be seen that after the client has an executable task, it sends the jar file containing the executable task to the Yarn scheduler; after the Yarn scheduler receives the jar file containing the executable task, it selects an idle computing node from the Spark cluster and sends the jar file containing the executable task to the idle computing node; the computing node parses the jar file, obtains the executable task, and executes the logic of the executable task.

[0051] However, there are special cases, that is, there are no idle computing nodes, but the executable tasks must be executed, so the executable tasks need to wait to be executed, that is, wait for idle computing nodes. Obviously, during the task processing, the Yarn scheduler needs to schedule idle computing nodes, and the idle computing nodes process the executable tasks. The scheduling process requires a certain amount of scheduling time; and when there are no idle computing nodes, the executable tasks are also required to wait to be processed, which increases the task processing time.

[0052] In view of this, and in response to the above problems, embodiments of the present application provide a method, device, and storage medium for task processing.

[0053] In this method, the driver node is pre-started and the message queue of executable tasks is detected. When there is an executable task sent by the client in the message queue, the executable task is read from the message queue as the target task, and the driver node executes the logic of the target task. The driver node is pre-selected from the Spark cluster by the Yarn scheduler based on the virtual task request sent by the client.

[0054] Therefore, when the client sends an executable task, the driver node reads the executable task and uses it as the target task, and directly executes the target task, skipping the scheduling process of the Yarn scheduler, reducing the probability of no idle driver nodes. This application optimizes time consumption and reduces the latency of task processing by saving the time of the Yarn scheduler selecting the driver node.

[0055] like Figure 2As shown, it is an application scenario diagram of a task processing provided by an embodiment of the present application. The server 20 is connected to a plurality of terminal devices 21 with clients installed via a network, and the network may be, but is not limited to, a local area network, a metropolitan area network, or a wide area network. The terminal device 21 may be a terminal device such as a personal computer (PC), a tablet computer, a personal digital assistant (PDA), a notebook, and a mobile phone, or a computer with a mobile terminal device, including various portable, pocket-sized, handheld, computer-built-in, or vehicle-mounted mobile devices, which can provide users with voice, data, or voice and data connectivity, and exchange voice, data, or voice and data with a wireless access network. The server 20 may be any background operation device that can provide Internet services and is used to manage stored data.

[0056] In this application scenario, the terminal device 21 is used to install and run the client. The client of the terminal device 21 is used to receive executable task instructions triggered by the user, determine the executable task, and send the executable task to the server 20. The server 20 executes the logic of the received executable task and feeds back the execution result corresponding to the executable task to the client of the terminal device 21.

[0057] In the present application, the client of the terminal device 21 receives the virtual task instruction and sends the virtual task instruction to the server 20. The server 20 pre-selects a computing node in the Spark cluster for executing the target task, and the pre-selected computing node detects in real time whether an executable task has arrived. When it is determined that an executable task has arrived, the executable task is used as the target task, and the computing node executes the target task, and feeds back the execution result of the target task to the client of the terminal device 21.

[0058] When the client of the terminal device 21 determines that the preset conditions are met, it actively sends a virtual task request to the server 20, so that the server 20 pre-selects a computing node in the Spark cluster for executing the target task, and the pre-selected computing node monitors in real time whether there is a target task arriving. When it is determined that the target task has arrived, the computing node executes the target task and returns the task execution result to the client of the terminal device 21.

[0059] This application scenario is mainly used in the client's data platform. The data platform uses the Spark distributed computing framework and integrates the Spark cluster and its usage interface. The data platform receives tasks triggered by users and completes task processing with the help of the Spark cluster. It is widely used in data analysis, training machine learning models, user credit rating, abnormal behavior detection, speech blocking and other fields.

[0060] For example, abnormal user behavior detection: Each user may have a large number of behaviors every day, and each behavior corresponds to a target task, such as login time, login location, purchase of props, gift of props, adding friends, changing user information, etc. Based on these behavioral data, detect whether the user account has been stolen, whether the user uses plug-ins, etc. Due to the large number of users and the large number of behaviors, it is difficult for one server to complete the analysis quickly, so it is necessary to use multiple servers for collaborative analysis to speed up the analysis. Multiple servers constitute a Spark cluster. How the Spark cluster divides the work to complete the abnormal user behavior detection needs to be managed using the Spark distributed computing framework.

[0061] For example, user speech blocking means blocking sensitive words and insulting words: users can speak and communicate in the game, but sensitive words and uncivilized language cannot be published. However, which sentences should be blocked requires machine learning training. Therefore, a large number of user speeches are collected, and the results of whether they should be blocked are manually marked for machine learning training. Due to the large amount of data, the training speed of a single machine is slow, and the joint training of multiple machines can accelerate it. Multiple machines form a Spark cluster, which provides a machine learning framework.

[0062] In this application, the client sends a virtual task request to the Yarn scheduler in the server. The Yarn scheduler selects idle computing nodes in the Spark cluster in advance according to the virtual task request, and uses the selected computing nodes as driver nodes. The driver node is pre-started, and the message queue is checked in real time to see if there are executable tasks sent by the client. When there are executable tasks sent by the client in the message queue, the executable task is read from the message queue and used as the target task, the logic of the target task is executed, and the execution result of the target task is sent to the message queue, so that the client can obtain the execution result of the target task from the message queue. Through the task processing method provided by this application, when a target task comes, the logic of the target task is directly executed through the driver node, which reduces the scheduling time of the Yarn scheduler. By saving the scheduling time of the Yarn scheduler, time consumption is optimized and the delay of task processing is reduced.

[0063] based on Figure 2 The application scenario is discussed below and the task processing method provided in the embodiment of the present application is introduced.

[0064] like Figure 3 As shown, it is a system architecture diagram of a task processing provided in an embodiment of the present application, wherein the system includes a client 30 and a server 31 , wherein the server 31 includes a Yarn scheduler 310 , a Spark cluster 311 , and a message queue 312 .

[0065] In the embodiment of task processing provided by the present application, in order to reduce the delay of task processing, when the target task arrives, the logic of the target task is directly executed by the driver node; instead of the Yarn scheduler starting to select the computing node for executing the target task when the target task arrives, and after the computing node for executing the target task is selected, the logic of the target task is executed by the selected computing node. The present application optimizes time consumption and reduces the delay of task processing by saving the scheduling time of the Yarn scheduler.

[0066] This application sends a virtual task request through the client and pre-starts the Yarn scheduler, so that the Yarn scheduler selects an idle computing node in the Spark cluster in advance, and uses the selected computing node as the driver node for executing the target task. The driver node pre-starts to detect whether there is an executable task sent by the client in the message queue; that is, the time for task startup and Yarn scheduler to select the computing node is moved forward; then the client sends the executable task to the message queue, and the pre-selected driver node reads the executable task from the message queue as the target task after detecting that there is an executable task in the message queue, and directly executes the logic of the target task to complete the processing process of the target task, etc., which saves the scheduling time of the Yarn scheduler and reduces the task processing delay.

[0067] Therefore, this application mainly consists of two parts:

[0068] 1. The client submits a virtual task to the Yarn scheduler, which selects idle computing nodes in the Spark cluster in advance and uses the selected computing nodes as driver nodes;

[0069] 2. The client sends the executable task to the message queue so that the driver node reads the executable task and feeds back the execution result of the executable task.

[0070] The present application is introduced through the following examples.

[0071] Embodiment 1: The client submits a virtual task to the Yarn scheduler, and the Yarn scheduler selects a computing node in the Spark cluster in advance.

[0072] like Figure 4 As shown, the present application provides a schematic diagram of sending a virtual task request and obtaining a driver node, including the following steps:

[0073] Step 400: The client sends a virtual task request to the Yarn scheduler.

[0074] When the client sends a virtual task request to the Yarn scheduler, it can be after receiving the instruction to send the virtual task request, and then send the virtual task request to the Yarn scheduler; or the client can automatically send the virtual task request to the Yarn scheduler after determining that a preset condition is met, wherein the preset condition can be that the number of driver nodes used to detect the message queue is less than a preset number.

[0075] Step 401: After receiving the virtual task request sent by the client, the Yarn scheduler allocates a driver node for executing the virtual task to the virtual task and sends the virtual task to the driver node.

[0076] In the present application, the Yarn scheduler sends a virtual task to the driver node, which is the Yarn scheduler sending a start instruction to the driver node to instruct the driver node to perform a pre-start.

[0077] After the Yarn scheduler receives the virtual task request sent by the client, it needs to allocate resources for the virtual task to execute the logic of the virtual task. At this time, the Yarn scheduler selects an idle computing node from the Spark cluster as a driver node to execute the virtual task. At this time, the driver node performs pre-start after receiving the request sent by the Yarn scheduler. Since the virtual task has no actual task code, the driver node waits for executable tasks. When an executable task arrives, the driver node directly executes the executable target task.

[0078] In this application, the client can send multiple virtual task requests to the Yarn scheduler at one time, so the Yarn scheduler needs to select multiple idle computing nodes in the Spark cluster for subsequent use.

[0079] After the driver node is selected, when the driver node is waiting for an executable task, it mainly detects and waits whether there is an executable task sent by the client in the message queue of the executable task.

[0080] Embodiment 2: Send an executable task, and let the driver node execute the logic of the executable task.

[0081] like Figure 5 As shown, the present application provides a schematic diagram of sending an executable task; comprising the following steps:

[0082] Step 500: The client sends the executable task to the message queue.

[0083] Since the executable tasks are transmitted to the driver node through the message queue, the executable tasks need to be serialized before being sent to the message queue.

[0084] The serialized executable task is described in the form of a json string. The following is an example of a serialized executable task provided in an embodiment of the present application:

[0085]

[0086]

[0087] It should be noted that the types of tasks that can be executed in this application include but are not limited to:

[0088] Spark SQL, compiled functions.

[0089] Step 501, the driver node detects the message queue of executable tasks in real time, and when it is detected that there is an executable task in the message queue, the driver node obtains the executable task from the message queue as a target task and executes the logic of the target task.

[0090] Since the executable tasks in the message queue are described in the form of JSON strings, when the driver node obtains the executable tasks from the message queue, it parses the executable tasks and executes the logic of the tasks according to the JSON description.

[0091] When the driver node executes the logic of the target task, the driver node determines whether other execution nodes are needed to cooperate in the execution according to the data volume of the target task to speed up the task processing.

[0092] Therefore, after the driver node obtains the target task, it analyzes the target task and obtains the data volume of the target task. When it detects that the data volume of the target task is greater than the preset data volume, the driver node sends a resource request to the Yarn scheduler to request the allocation of other execution nodes. The Yarn scheduler selects an idle computing node from the Spark cluster and returns the computing node as an execution node to the driver node.

[0093] The driver node distributes the target task to the execution node returned by the Yarn scheduler. The driver node and the execution node jointly execute the logic of the target task in a parallel execution mode to speed up task processing.

[0094] In this application, after the driver node completes the target task, it feeds back the execution result corresponding to the target task to the client. Figure 6 As shown, it is a schematic diagram of a driving node feedback execution result provided in an embodiment of the present application; it includes the following steps:

[0095] Step 600: The driver node sends the execution result of the target task to the message queue.

[0096] The task execution results include but are not limited to:

[0097] Task processing success, task processing failure, task exception.

[0098] Step 601: The client detects the message queue in real time, and when detecting the execution result in the message queue, obtains the execution result from the message queue.

[0099] In the present application, when the client and the driver node transmit executable tasks and execution results through the message queue, the executable tasks and the execution results corresponding to the executable tasks are obtained through UUID (Universally Unique Identifier).

[0100] Specifically, when an executable task arrives on the client, a UUID is generated and added to the UUID list in the message queue, and then the executable task is added to the container indexed by the UUID. The driver node obtains the executable task based on the UUID, executes the logic of the executable task, determines the execution result, and feeds the execution result back to the container indexed by the UUID corresponding to the executable task in the message queue. The client reads the execution result based on the UUID. The same UUID is used to send executable tasks and read execution results to prevent confusion caused by multiple clients requiring task processing.

[0101] like Figure 7 The figure is a schematic diagram of the overall process of task processing provided by an embodiment of the present application. Figure 7 It can be seen that the client first sends a virtual task request to the Yarn scheduler to pre-start the Yarn scheduler; the Yarn scheduler sends the virtual task request to the Spark cluster and selects an idle computing node as the driver node; then the client sends the executable task to the message queue, and the driver node reads the executable task in the message queue as the target task and executes the logic of the target task; the driver node sends the execution result of the target task to the message queue, and the client obtains the execution result from the message queue. In the existing technical solution, the target task is sent directly to the Yarn scheduler. After receiving the target task, the Yarn scheduler selects an idle computing node as the driver node, and the driver node executes the logic of the target task.

[0102] like Figure 8 The figure is a schematic diagram of the effect of comparing the technical solution provided by the embodiment of the present application with the existing related technical solution in terms of time when performing task processing. Figure 8 It can be seen that the time for task processing in the present application is shorter than that in the existing related technical solutions. Therefore, the technical solution provided in the embodiment of the present application reduces the delay of task processing.

[0103] Based on the same inventive concept, the embodiment of the present application provides a method for task processing, such as Fig. 9 As shown, it is a flowchart of a task processing method provided in an embodiment of the present application, comprising the following steps:

[0104] Step 900: The driver node performs pre-startup according to the start-up instruction of the Yarn scheduler, wherein the driver node is selected from the Spark cluster by the Yarn scheduler according to the virtual task request sent by the client;

[0105] Step 901: The driver node obtains a target task from a message queue of executable tasks and executes the obtained target task, wherein the executable task in the message queue is sent by the client;

[0106] Step 902: The driver node sends the execution result of the target task to the message queue, so that the client obtains the execution result of the target task from the message queue.

[0107] In a possible implementation, the target task acquired by the driver node includes:

[0108] The driver node analyzes the target task;

[0109] When the data type of the target task is a distributed dataset RDD data type, the driver node distributes the target task to the execution node, so that the driver node and the execution node jointly execute the target task;

[0110] The execution node is obtained by sending a resource request to the Yarn scheduler after the driver node determines that the data type of the target task is the RDD data type.

[0111] In a possible implementation, the driver node obtains the target task from the message queue of the executable task, including:

[0112] The driver node detects the message queue of executable tasks in real time;

[0113] When there is an executable task in the message queue of the executable task, the driver node obtains the executable task as the target task.

[0114] Based on the same inventive concept, the embodiment of the present application also provides another task processing method, such as Fig.10 As shown, it is a flowchart of a task processing method provided in an embodiment of the present application, comprising the following steps:

[0115] Step 1000, the client sends the executable task as a target task to the message queue, so that the driver node obtains the target task from the message queue;

[0116] Step 1001: The client detects the message queue and receives the execution result of the target task fed back by the driver node.

[0117] In a possible implementation, before the client sends the executable task as a target task to the message queue, the client serializes the executable task.

[0118] In a possible implementation, before the client sends the executable task as the target task to the message queue, the client sends a virtual task request to the Yarn scheduler, so that the Yarn scheduler selects a driver node for detecting the message queue from the Spark cluster.

[0119] Based on the same inventive concept, the embodiment of the present application also provides a task processing device 1100, such as Fig.11 As shown, the device 1100 includes: a pre-starting unit 1101, an execution unit 1102 and a first sending unit 1103; wherein:

[0120] A pre-start unit 1101 is used to perform pre-start according to a start instruction of a Yarn scheduler, wherein a driver node is selected from a Spark cluster by the Yarn scheduler according to a virtual task request sent by a client;

[0121] An execution unit 1102 is used to obtain a target task from a message queue of executable tasks and execute the obtained target task, wherein the executable task in the message queue is sent by the client;

[0122] The first sending unit 1103 is used to send the execution result of the target task to the message queue, so that the client obtains the execution result of the target task from the message queue.

[0123] In a possible implementation, the execution unit 1102 is further configured to: detect a message queue of executable tasks in real time; and when there is an executable task in the message queue of executable tasks, obtain the executable task as a target task.

[0124] In a possible implementation, the execution unit 1102 is further used to: analyze the target task; when the data type of the target task is a distributed data set RDD data type, distribute the target task to the execution node, so that the driver node and the execution node jointly execute the target task;

[0125] The execution node is obtained by sending a resource request to the Yarn scheduler after determining that the data type of the target task is the RDD data type.

[0126] Based on the same inventive concept, the embodiment of the present application also provides a task processing device 1200, such as Fig.12 As shown, the device 1200 includes: a second sending unit 1201 and a receiving unit 1202, wherein:

[0127] The second sending unit 1201 is used to send the executable task as a target task to the message queue, so that the driver node obtains the target task from the message queue;

[0128] The receiving unit 1202 is used to detect the message queue and receive the execution result of the target task fed back by the driving node.

[0129] In a possible implementation, the second sending unit 1201 is further configured to: perform serialization processing on the executable tasks.

[0130] In a possible implementation, the second sending unit 1201 is further configured to: send a virtual task request to the Yarn scheduler, so that the Yarn scheduler selects a driver node for detecting a message queue from the Spark cluster.

[0131] For the convenience of description, the above parts are divided into units (or modules) according to their functions and described separately. Of course, when implementing this application, the functions of each unit (or module) can be implemented in the same or multiple software or hardware.

[0132] After introducing the method and apparatus for task processing according to an exemplary embodiment of the present application, a computing device for task processing according to another exemplary embodiment of the present application is introduced next.

[0133] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0134] In a possible implementation, the task processing computing device provided in the embodiment of the present application may include at least a processor and a memory. The memory stores a program code, and when the program code is executed by the processor, the processor executes any step of the task processing method in various exemplary embodiments of the present application.

[0135] Refer to the following Fig.13 The task processing computing device 1300 according to this embodiment of the present application is described as follows. Fig.13 The task processing computing device 1300 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0136] like Fig.13As shown, the components of the task processing computing device 1300 may include but are not limited to: the at least one processor 1301 mentioned above, the at least one memory 1302 mentioned above, and a bus 1303 connecting different system components (including the memory 1302 and the processor 1301).

[0137] Bus 1303 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0138] The memory 1302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 13021 and / or a cache memory 13022 , and may further include a read-only memory (ROM) 13023 .

[0139] Memory 1302 may also include a program / utility 13025 having a set (at least one) of program modules 13024, such program modules 13024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0140] The task processing computing device 1300 may also communicate with one or more external devices 1304 (e.g., keyboards, pointing devices, etc.), one or more devices that enable a user to interact with the task processing computing device 1300, and / or any device that enables the task processing computing device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1305. Furthermore, the task processing computing device 1300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1306. Fig.13 As shown, the network adapter 1306 communicates with other modules of the task processing computing device 1300 via the bus 1303. It should be understood that although Fig.13 Not shown, other hardware and / or software modules may be used in conjunction with the task processing computing device 1300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0141] In some possible implementations, various aspects of the task processing method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the task processing method according to various exemplary implementations of the present application described above in this specification.

[0142] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] The program product generated by the task processing of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, device or device.

[0144] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0145] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0146] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user equipment, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0147] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0148] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0154] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A task processing method, characterized in that: The method includes: The driver node performs pre-startup according to the startup instruction of the Yarn scheduler, wherein the driver node is an idle computing node selected by the Yarn scheduler from the Spark cluster according to the virtual task request sent by the client; the startup instruction is the virtual task in the virtual task request sent by the Yarn scheduler to the driver node; The driver node detects the message queue of executable tasks in real time, and when detecting that there is an executable target task in the message queue, obtains the target task from the message queue of executable tasks, and executes the obtained target task, wherein the executable task in the message queue is sent by the client; The driver node sends the execution result of the target task to the message queue, so that the client obtains the execution result of the target task from the message queue.

2. The method according to claim 1, characterized in that The driver node executes the acquired target task, including: The driving node analyzes the target task; When the data type of the target task is a distributed data set RDD data type, the driver node distributes the target task to the execution node, so that the driver node and the execution node jointly execute the target task; The execution node is obtained by sending a resource request to the Yarn scheduler after the driver node determines that the data type of the target task is the RDD data type.

3. A task processing method, characterized in that: The method includes: The client sends the executable task as the target task to the message queue, so that the driver node detects the message queue of the executable task in real time. When the driver node detects that there is an executable target task in the message queue, it obtains the target task from the message queue. The client detects the message queue and receives the execution result of the target task fed back by the driver node; Wherein, the method further comprises: The client sends a virtual task request to the Yarn scheduler, so that the Yarn scheduler selects an idle computing node for detecting a message queue from the Spark cluster, and uses the selected computing node as a driver node, and sends the virtual task in the virtual task request to the driver node, so that the driver node instructs the execution of pre-start.

4. The method according to claim 3, characterized in that Before the client sends the executable task as the target task to the message queue, the method further includes: The client serializes the executable tasks.

5. A task processing device, characterized in that: The device comprises: a pre-starting unit, an execution unit and a first sending unit; wherein: The pre-start unit is used to perform pre-start according to the start instruction of the Yarn scheduler, wherein the driver node is an idle computing node selected by the Yarn scheduler from the Spark cluster according to the virtual task request sent by the client; the start instruction is the virtual task in the virtual task request sent by the Yarn scheduler to the driver node; The execution unit is used to detect the message queue of executable tasks in real time, and when it is detected that there is an executable target task in the message queue, obtain the target task from the message queue of executable tasks and execute the obtained target task, wherein the executable task in the message queue is sent by the client; The first sending unit is used to send the execution result of the target task to the message queue, so that the client obtains the execution result of the target task from the message queue.

6. A task processing device, characterized in that: The device comprises: a second sending unit and a receiving unit, wherein: The second sending unit is used to send the executable task as the target task to the message queue, so that the driver node obtains the target task from the message queue; The receiving unit is used to detect the message queue and receive the execution result of the target task fed back by the driving node; The second sending unit is also used to: send a virtual task request to the Yarn scheduler, so that the Yarn scheduler selects an idle computing node for detecting the message queue from the Spark cluster, and uses the selected computing node as a driver node, and sends the virtual task in the virtual task request to the driver node, so that the driver node instructs to execute pre-start.

7. A computing device, characterized in that: The invention comprises at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 2 or claims 3 to 4.

8. A computer-readable medium, characterized in that It stores a computer program executable by a computing device, and when the program is run on the computing device, the computing device executes the steps of the method described in any one of claims 1 to 2 or claims 3 to 4.

Images