Task processing method and device, computer equipment and computer readable storage medium

Abstract

The invention provides a task processing method and device, computer equipment and a computer readable storage medium, and the method comprises the steps: receiving task processing information of a Spark task, the task processing information comprising a task processing node blacklist; determining a plurality of first task processing nodes required for processing the Spark task according to the task processing information; verifying each first task processing node based on the task processing node blacklist; determining the first task processing node passing the verification as a second task processing node, and determining a task process corresponding to the second task processing node according to the task processing information; and executing the task process based on the second task processing node. According to the method, the calling flexibility of the task processing node and the processing efficiency of the Spark task are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a task processing method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Currently, when using spark tasks to process large amounts of data, it is necessary to fully call and apply the computing resources of large-scale clusters to complete data processing. Therefore, most of the current methods rely on the cluster's scheduling manager to call multiple task processing nodes in the cluster and distribute data to multiple task processing nodes to achieve the allocation of computing resources. However, most of the current allocation methods use whether the available resources on the task processing nodes are sufficient as the allocation principle, and the allocation method is relatively simple. In addition, the current allocation method is prone to change failure when the task processing node changes the task type to be processed, resulting in low processing efficiency of spark tasks. Summary of the invention

[0003] The present application provides a task processing method, apparatus, computer equipment and storage medium, aiming to improve the efficiency and flexibility of processing spark tasks.

[0004] In a first aspect, the present application provides a task processing method, the task processing method comprising the following steps:

[0005] Receive task processing information of a Spark task, wherein the task processing information includes a task processing node blacklist;

[0006] Determine, according to the task processing information, a plurality of first task processing nodes required to process the Spark task;

[0007] Verifying each of the first task processing nodes based on the task processing node blacklist;

[0008] Determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information;

[0009] The task process is executed based on the second task processing node.

[0010] In a second aspect, the present application further provides a task processing device, the task processing device comprising:

[0011] A task receiving module, used to receive task processing information of a Spark task, wherein the task processing information includes a task processing node blacklist;

[0012] A first node determination module, used to determine a plurality of first task processing nodes required to process the Spark task according to the task processing information;

[0013] A node verification module, used for verifying each of the first task processing nodes based on the task processing node blacklist;

[0014] A second node determination module, used to determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information;

[0015] A task processing module is used to execute the task process based on the second task processing node.

[0016] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the task processing method as described above are implemented.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the task processing method as described above are implemented.

[0018] The present application provides a task processing method, apparatus, computer equipment and computer-readable storage medium. The present application determines a plurality of first task processing nodes according to task processing information, verifies each first task processing node based on a task processing node blacklist, determines the first task processing node that passes the verification as a second task processing node, determines a task process corresponding to the second task processing node according to the second task processing node, executes the task process based on the second task processing node to complete the processing of the Spark task, thereby realizing the customization of the task processing node, improving the flexibility of mobilizing Spark task resources, and avoiding the allocation of the task process of the Spark task to the task processing node that needs to change the processing task, which causes an error in the task processing node, thereby improving the flexibility of calling the task processing node and the processing efficiency of the Spark task. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a task processing method provided in an embodiment of the present application;

[0021] Figure 2 A scene diagram of a task processing method provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a task processing device provided by an embodiment of the present application;

[0023] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] The embodiments of the present application provide a task processing method, apparatus, computer equipment and computer-readable storage medium. The task processing method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, etc. It can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A flowchart of a task processing method provided in one embodiment of the present application.

[0030] like Figure 1As shown, the task processing method includes steps S101 to S105.

[0031] Step S101: Receive task processing information of a Spark task, where the task processing information includes a task processing node blacklist.

[0032] Exemplarily, Spark tasks include but are not limited to big data processing scenarios such as data query, offline computing, data ERL processing, data mining and machine learning. Specifically, for example, a company's group insurance policy data query, it should be understood that the company's group insurance policy includes but is not limited to the insurance policy of each employee in the company, the insurance policy of items, and the insurance policy of property, etc. Therefore, more data is involved. In the process of data query, a Spark task can be established to realize the query of such data.

[0033] In the specific implementation process, the user establishes a Spark task at the Spark task initiator and determines a blacklist of processing task nodes, so that when the task receiver receives the Spark task, it can determine the blacklist of processing task nodes, thereby processing the Spark task based on the processing task nodes outside the blacklist of processing task nodes, so that the user can customize the configuration of the processing task nodes.

[0034] Step S102: Determine a plurality of first task processing nodes required for processing the Spark task according to the task processing information.

[0035] Exemplarily, after obtaining the task processing information of the Spark task, multiple first task processing nodes for processing the Spark task are determined according to the task processing information. It should be understood that by distributing the Spark task to multiple first task processing nodes, the Spark task can be processed based on multiple computing resources to provide computing power for big data calculations and improve the processing efficiency of the Spark task.

[0036] It should be understood that in addition to the task processing nodes used to process Spark tasks, task processing nodes for processing other tasks are also included, and the task processing nodes can be called to process different tasks. That is, if an idle task processing node is called to process a Spark task, the task processing node is determined as the first task processing node. In the specific implementation process, if the first task processing node completes the processing of the Spark task and is in an idle state, the first task processing node is adjusted to a task processing node for processing other tasks to realize the calling of the first task processing node to process other tasks. This application does not limit the specific calling method of the task processing node.

[0037] Step S103: verify each of the first task processing nodes based on the task processing node blacklist.

[0038] Exemplarily, it is determined whether the first task processing node is in a task processing node blacklist to implement the selection of the task processing node.

[0039] In some embodiments, the method further includes: if the task processing node blacklist does not include the first task processing node currently being verified, determining that the first task processing node currently being verified has passed the verification; if the task processing node blacklist includes the first task processing node currently being verified, determining that the first task processing node currently being verified has not passed the verification.

[0040] Exemplarily, if the first task processing node is in the task processing blacklist, it is determined that the first task processing node has failed the verification, otherwise it has passed the verification, so that the user can customize the task processing node for processing the Spark task.

[0041] In some embodiments, the verifying each of the first task processing nodes based on the task processing node blacklist includes: verifying each of the first task processing nodes based on a driver process generated in a NodeManager of a Yarn cluster.

[0042] See also Figure 2 , Figure 2 This is a scene diagram of a task processing method provided in an embodiment of the present application. Figure 2 As shown, for ease of understanding, the following embodiments are described using the Yarn cluster as an example.

[0043] Exemplarily, the first task processing node is verified based on the NodeManager in the Yarn cluster. Specifically, after receiving the task processing node blacklist and the first task processing node, the NodeManager generates a diver process to verify the first task processing node based on the task processing node blacklist to determine the first task processing node that cannot execute the Spark task. It should be understood that generating a diver process in the NodeManager to verify the first task processing node can directly screen and eliminate the task processing nodes without rebuilding the Yarn cluster based on the task processing node blacklist, thereby improving the processing efficiency of the Spark task.

[0044] In some embodiments, determining the multiple first task processing nodes required to process the Spark task according to the task processing information includes: based on the ResourceManager of the Yarn cluster, determining the multiple first task processing nodes required to process the Spark task according to the task information, and transmitting the first task processing nodes and the task processing node blacklist to the NodeManager.

[0045] Exemplarily, the ResourceManager in the Yarn cluster receives the task processing information sent from the Spark task initiator. After receiving the task processing information, the ResourceManager determines the multiple first task processing nodes required to process the corresponding Spark task based on the task processing information, and transmits the determined first task processing nodes and the task processing node blacklist to the NodeManager, so that the NodeManager verifies the received first task processing nodes based on the received task processing node blacklist. Specifically, the ResourceManager does not verify the first task processing node, but only determines the required resources based on the task processing information, and the verification process is completed in the NodeManager. It should be understood that the Spark task contains multiple task processes, and the multiple task processes can be assigned to different first task processing nodes for processing to improve task processing efficiency.

[0046] Step S104: determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information.

[0047] In a specific implementation process, NodeManager determines the first task processing node that passes the verification as the second task processing node, and determines the task process corresponding to the second task processing node to allocate the task process to the corresponding second task processing node. It should be understood that the second task processing node can also be NodeManager.

[0048] In some embodiments, the method further includes: based on the NodeManager, deleting the first task processing nodes that have not passed the verification, so as to use the remaining first task processing nodes as the second task processing nodes.

[0049] Exemplarily, in NodeManager, the first task processing node that fails the verification is deleted to prevent the first task processing node that fails the verification from processing the corresponding task process, and the first task processing node that has not been deleted is determined as the second task processing node, thereby implementing the screening of task processing nodes to prevent the first task processing node in the task processing node blacklist from processing the Spark task.

[0050] In the specific implementation process, after deleting the first task processing node in the task processing node blacklist and obtaining the second task processing node, NodeManager generates multiple task processes according to the task processing information and distributes the task processes to each second task processing node to realize task distribution processing.

[0051] Step S105: executing the task process based on the second task processing node.

[0052] Exemplarily, after receiving the assigned task process, the second task processing node processes the task process. It should be understood that when each second task processing node completes processing the assigned task process, the corresponding Spark task is also completed, thereby realizing the processing of Spark tasks while utilizing multiple resources, such as multiple second task processing nodes, to realize task processing, thereby improving task processing efficiency, and the second task processing nodes are all task processing nodes outside the task processing blacklist, realizing user-defined task processing nodes.

[0053] It should be noted that in the existing specific application scenarios of calling Yarn clusters, users often preset high and low peaks to flexibly schedule the computing resources of task processing nodes. For example, some task processing nodes are configured to be expanded to Yarn's NodeManager during the daytime peak period to allocate resources to T+1 type batch computing tasks. When such computing tasks are in the off-peak period at night, the NodeManager service on the task processing node expanded to NodeManager will be removed to return the task processing node to the OLAP service, thereby improving the cluster's operating efficiency and reducing usage costs. However, if the task processing node extended to Yarn's NodeManager processes Spark tasks, the existing architecture cannot support ensuring that the task processing node can complete the corresponding task process processing or avoid allocating the task process to the task processing node before removing the NodeManager service, resulting in the task processing node being unable to remove the NodeManager service because it is still processing the corresponding task process, thereby failing to change the service. The present application sets a task processing node blacklist at the Spark task initiator. For example, when the task processing node is preparing to change the service, the task processing node is written into the task processing node blacklist, so that the task processing node does not need to execute the Spark task, thereby avoiding the situation where the task processing node cannot change the service due to being assigned a Spark task again when the service needs to be changed, thereby realizing flexible adjustment of the cluster.

[0054] In some embodiments, the method also includes: in the event that the task process cannot be processed by the second task processing node, determining a process processing request based on the processing status of the task processing information; determining a third task processing node based on the process processing request, and verifying the third task processing node based on the task processing node blacklist; in the event that the third task processing node passes the verification, processing the task process that the second task processing node cannot process based on the third task processing node.

[0055] Exemplarily, in determining the task process, it is determined according to the process that the second task processing node can process, so that the second task processing node can process all task processes corresponding to the Spark task. However, in a specific implementation, the task process and the first task processing node are both determined in ResourceManager, and there is a corresponding relationship between the task process and the first task processing node, and because ResourceManager does not verify the first task processing node, after verifying the first task processing node in NodeManager, the first task processing node that is the same as the task processing node in the task processing node blacklist will be deleted, resulting in the number of second task processing nodes obtained being less than the number of first task processing nodes, resulting in some task processes having no corresponding second task processing nodes; in this case, if the task process and the first task processing node are in a one-to-many or many-to-many relationship, then after a first task processing node corresponding to the task process is deleted, it can also be determined whether there are other corresponding task processing nodes for the task process, and if there are and the task processing node is a second task processing node, the task process is assigned to the corresponding second task processing node.

[0056] In other embodiments, if the task process and the task processing node are one-to-one and the first task processing node corresponding to the task process is deleted, or the task process and the first task processing node are one-to-many, all first task processing nodes corresponding to the task process are deleted; the NodeManager can determine the second task processing node that can handle these uncorresponding task processes among multiple second task processing nodes, and assign these uncorresponding task processes to the second task nodes that can handle them for processing; and if the NodeManager determines that all current second task processing nodes cannot handle these task processes, it generates a process processing request based on the task processes that cannot be processed, and transmits the process processing request to the ResourceManager, so that the ResourceManager re-determines the task processing node according to the process processing request, so that the re-determined task processing node can handle these unprocessed task processes.

[0057] In other embodiments, NodeManager is unable to determine whether the second task processing node is capable of processing these task processes, that is, NodeManager does not have the function of determining the task processing node used to process the task process based on the task process. Therefore, after NodeManager deletes the first task processing node and the task process has no corresponding task processing node, it directly generates a process processing request based on these task processes that have no corresponding relationship, and transmits the process processing request to ResourceManager, so that ResourceManager determines the task processing node for processing these task processes again based on the process processing request.

[0058] Specifically, ResourceManager determines the third task processing node according to the process processing request, and transmits the third task processing node to NodeManager. Since ResourceManager will not verify the third task processing node, the third task processing node may still be in the task processing node blacklist. Therefore, after receiving the third task processing node, NodeManager verifies the third task processing node based on the task processing node blacklist. If the third task processing node passes the verification, the task process that just failed to be allocated is allocated to the third task processing node, so that the third task processing node can process the task process that the second task processing node failed to process.

[0059] In some embodiments, the method further includes: if the third task processing node fails verification, re-determining the task processing node according to a verification result that the third task processing node fails verification and the process processing request.

[0060] During the specific implementation process, if the third task processing node fails the verification, the NodeManager will generate a process processing request again based on the verification result of the third task processing node failing the verification and the task process that failed to be processed, and transmit the process processing request to the ResourceManager, so that the ResourceManager will re-determine the task processing node based on the received process processing request. It should be understood that the re-determined task processing node does not include the third task processing node determined last time, so as to avoid repeated verification of the same task node.

[0061] The task processing method provided in the above embodiment implements, through the task processing node blacklist set by the user, that in the application scenario of Spark tasks, the user can customize the task processing nodes, so that the resource scheduling of Spark tasks is more flexible, and it can avoid allocating the task process of the Spark task to the task processing node that needs to change the processing task, which causes an error in the task processing node, and when the second task processing node after verification fails to meet all the task processes of the Spark task, it can re-determine the task processing node that processes the task process, and it can ensure that the re-determined task processing node is no longer in the task processing node blacklist, avoiding errors in the task processing node when it becomes a service, and improving the calling flexibility of the task processing node and the processing efficiency of the Spark task.

[0062] See also Figure 3 , Figure 3 is a schematic diagram of a task processing device provided in an embodiment of the present application. The task processing device can be configured in a server or a terminal to execute the aforementioned task processing method.

[0063] like Figure 3 As shown, the task processing device 100 includes: a task receiving module 110, a first node determination module 120, a node verification module 130, a second node determination module 140, and a task processing module 150.

[0064] The task receiving module 110 is used to receive task processing information of a Spark task, where the task processing information includes a task processing node blacklist.

[0065] The first node determination module 120 is used to determine a plurality of first task processing nodes required to process the Spark task according to the task processing information.

[0066] The node verification module 130 is used to verify each of the first task processing nodes based on the task processing node blacklist.

[0067] The second node determination module 140 is used to determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information.

[0068] The task processing module 150 is used to execute the task process based on the second task processing node.

[0069] Exemplarily, the node verification module 130 includes a NodeManager submodule, a text repetition rate calculation submodule, a second text determination submodule, a text deletion submodule, and a text matching submodule.

[0070] The NodeManager submodule is used to generate a driver process to verify each of the first task processing nodes based on the driver process.

[0071] Exemplarily, the NodeManager submodule is also used to determine that the first task processing node currently being verified has passed the verification if the task processing node blacklist does not include the first task processing node currently being verified; if the task processing node blacklist includes the first task processing node currently being verified, determine that the first task processing node currently being verified has not passed the verification.

[0072] Exemplarily, the first node determination module 120 includes a ResourceManager submodule.

[0073] The ResourceManager submodule is used to determine a plurality of first task processing nodes required to process the Spark task according to the task information, and transmit the first task processing nodes and the task processing node blacklist to the NodeManager submodule.

[0074] Exemplarily, the NodeManager submodule is further configured to delete the first task processing nodes that have not passed the verification, so as to use the remaining first task processing nodes as the second task processing nodes.

[0075] Exemplarily, the task processing device 100 further includes a request generation submodule.

[0076] The request generation submodule is used to determine a process processing request according to the processing status of the task processing information when the task process cannot be processed by the second task processing node.

[0077] The ResourceManager submodule is further configured to determine a third task processing node according to the process processing request.

[0078] The NodeManager submodule is further used to verify the third task processing node according to the task processing node blacklist.

[0079] The task processing module 150 is further configured to process the task process that cannot be processed by the second task processing node based on the third task processing node when the third task processing node passes the verification.

[0080] Exemplarily, the NodeManager submodule is also used to regenerate a process processing request based on the verification result of the third task processing node failing the verification and the process processing request if the third task processing node fails the verification, and transmit the regenerated process processing request to the ResourceManager submodule, so that the ResourceManager submodule re-determines the task processing node.

[0081] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0082] See also Figure 4 , Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0083] like Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0084] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any task processing method.

[0085] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0086] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any task processing method.

[0087] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0088] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0089] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0090] Receive task processing information of a Spark task, wherein the task processing information includes a task processing node blacklist;

[0091] Determine, according to the task processing information, a plurality of first task processing nodes required to process the Spark task;

[0092] Verifying each of the first task processing nodes based on the task processing node blacklist;

[0093] Determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information;

[0094] The task process is executed based on the second task processing node.

[0095] In one embodiment, when the processor verifies each of the first task processing nodes based on the task processing node blacklist, it is configured to implement:

[0096] The first task processing nodes are verified based on the driver process generated in the NodeManager of the Yarn cluster.

[0097] In one embodiment, when implementing the task processing method, the processor is used to implement:

[0098] If the task processing node blacklist does not include the first task processing node currently being verified, determining that the first task processing node currently being verified passes verification;

[0099] If the task processing node blacklist includes the first task processing node currently being verified, it is determined that the first task processing node currently being verified has failed verification.

[0100] In one embodiment, when determining a plurality of first task processing nodes required to process the Spark task according to the task processing information, the processor is configured to implement:

[0101] Based on the ResourceManager of the Yarn cluster, a plurality of first task processing nodes required for processing the Spark task are determined according to the task information, and the first task processing nodes and the task processing node blacklist are transmitted to the NodeManager.

[0102] In one embodiment, when implementing the task processing method, the processor is used to implement:

[0103] Based on the NodeManager, the first task processing nodes that fail the verification are deleted, so that the remaining first task processing nodes are used as the second task processing nodes.

[0104] In one embodiment, when implementing the task processing method, the processor is used to implement:

[0105] In the case where the task process cannot be processed by the second task processing node, determining a process processing request according to a processing status of the task processing information;

[0106] Determine a third task processing node according to the process processing request, and verify the third task processing node according to the task processing node blacklist;

[0107] In the case that the third task processing node passes the verification, the task process that the second task processing node cannot process is processed based on the third task processing node.

[0108] In one embodiment, when implementing the task processing method, the processor is used to implement:

[0109] If the third task processing node fails the verification, the task processing node is re-determined according to the verification result that the third task processing node fails the verification and the process processing request.

[0110] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned task processing method embodiment, and will not be repeated here.

[0111] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the task processing method of the present application.

[0112] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0113] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0114] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0115] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A task processing method, characterized in that: include: Receive task processing information of a Spark task, wherein the task processing information includes a task processing node blacklist; Determine, according to the task processing information, a plurality of first task processing nodes required to process the Spark task; Verifying each of the first task processing nodes based on the task processing node blacklist; Determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information; The task process is executed based on the second task processing node.

2. The task processing method according to claim 1, characterized in that: The verifying each of the first task processing nodes based on the task processing node blacklist includes: The first task processing nodes are verified based on the driver process generated in the NodeManager of the Yarn cluster.

3. The task processing method according to claim 2, characterized in that: The method further comprises: If the task processing node blacklist does not include the first task processing node currently being verified, determining that the first task processing node currently being verified passes verification; If the task processing node blacklist includes the first task processing node currently being verified, it is determined that the first task processing node currently being verified has failed verification.

4. The task processing method according to claim 2, characterized in that: The determining, according to the task processing information, a plurality of first task processing nodes required for processing the Spark task comprises: Based on the ResourceManager of the Yarn cluster, a plurality of first task processing nodes required for processing the Spark task are determined according to the task information, and the first task processing nodes and the task processing node blacklist are transmitted to the NodeManager.

5. The task processing method according to claim 4, characterized in that: The method further comprises: Based on the NodeManager, the first task processing nodes that fail the verification are deleted, so that the remaining first task processing nodes are used as the second task processing nodes.

6. The task processing method according to any one of claims 1 to 5, characterized in that: The method further comprises: In the case where the task process cannot be processed by the second task processing node, determining a process processing request according to a processing status of the task processing information; Determine a third task processing node according to the process processing request, and verify the third task processing node according to the task processing node blacklist; In the case that the third task processing node passes the verification, the task process that the second task processing node cannot process is processed based on the third task processing node.

7. The task processing method according to claim 6, characterized in that: The method further comprises: If the third task processing node fails the verification, the task processing node is re-determined according to the verification result that the third task processing node fails the verification and the process processing request.

8. A task processing device, characterized in that: The task processing device comprises: A task receiving module, used to receive task processing information of a Spark task, wherein the task processing information includes a task processing node blacklist; A first node determination module, used to determine a plurality of first task processing nodes required to process the Spark task according to the task processing information; A node verification module, used for verifying each of the first task processing nodes based on the task processing node blacklist; A second node determination module, used to determine the first task processing node that passes the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information; A task processing module is used to execute the task process based on the second task processing node.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the task processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the task processing method according to any one of claims 1 to 7 are implemented.

Images