Task processing method and device, computer device and computer readable storage medium

Abstract

The application provides a task processing method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: receiving task processing information of a Spark task, wherein the task processing information comprises a task processing node blacklist; determining a plurality of first task processing nodes required for processing the Spark task according to the task processing information; checking each first task processing node based on the task processing node blacklist; determining a second task processing node from the first task processing nodes that pass the check, and determining a task process corresponding to the second task processing node according to the task processing information; and executing the task process based on the second task processing node. The application improves the calling flexibility of the task processing node and the processing efficiency of the Spark task.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a task processing method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] Currently, when using Spark tasks to process large amounts of data, it is necessary to fully utilize the computing resources of large-scale clusters to complete the data processing. Therefore, most current methods rely on the cluster's scheduler to call multiple task processing nodes in the cluster and allocate data to multiple task processing nodes to achieve the allocation of computing resources. However, most current allocation methods are based on whether the available resources on the task processing nodes are sufficient, resulting in a relatively simple allocation method. Furthermore, the current allocation method is prone to failure when the task processing node changes the type of task it is processing, leading to low processing efficiency of Spark tasks. Summary of the Invention

[0003] This application provides a task processing method, apparatus, computer device, and storage medium, aiming to improve the efficiency and flexibility of processing Spark tasks.

[0004] Firstly, this application provides a task processing method, which includes the following steps:

[0005] Receive task processing information for Spark tasks, including a blacklist of task processing nodes;

[0006] Based on the task processing information, determine the multiple first task processing nodes required to process the Spark task;

[0007] Verify each of the first task processing nodes based on the task processing node blacklist;

[0008] The first task processing node that passes the verification is determined as the second task processing node, and the task process corresponding to the second task processing node is determined according to the task processing information.

[0009] The task process is executed based on the second task processing node.

[0010] Secondly, this application also provides a task processing apparatus, the task processing apparatus comprising:

[0011] The task receiving module is used to receive task processing information for Spark tasks, including a blacklist of task processing nodes.

[0012] The first node determination module is used to determine multiple first task processing nodes required to process the Spark task based on the task processing information.

[0013] The node verification module is used to verify each of the first task processing nodes based on the task processing node blacklist.

[0014] The second node determination module is used to determine the first task processing node that has passed the verification as the second task processing node, and to determine the task process corresponding to the second task processing node according to the task processing information.

[0015] The task processing module is used to execute the task process based on the second task processing node.

[0016] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the task processing method described above.

[0017] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the task processing method described above.

[0018] This application provides a task processing method, apparatus, computer device, and computer-readable storage medium. The method involves determining multiple first task processing nodes based on task processing information, verifying each first task processing node based on a task processing node blacklist, identifying the verified first task processing node as a second task processing node, determining the corresponding task process based on the second task processing node, and executing the task process based on the second task processing node to complete the processing of the Spark task. This achieves customization of task processing nodes, improves the flexibility of Spark task resource allocation, and avoids errors caused by assigning Spark task processes to task processing nodes that require changes in processing, thereby improving the flexibility of calling task processing nodes and the processing efficiency of Spark tasks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a task processing method provided in an embodiment of this application;

[0021] Figure 2 A scenario diagram illustrating a task processing method provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a task processing device provided in an embodiment of this application;

[0023] Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0027] This application provides a task processing method, apparatus, computer device, and computer-readable storage medium. The task processing method can be applied to a terminal device, such as a tablet computer, laptop computer, or desktop computer. It can also be applied to a server, which can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0028] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a task processing method provided in one embodiment of this application.

[0030] like Figure 1As shown, the task processing method includes steps S101 to S105.

[0031] Step S101: Receive task processing information for Spark tasks, including a blacklist of task processing nodes.

[0032] For example, Spark tasks include, but are not limited to, big data processing scenarios such as data querying, offline computing, data ERL processing, data mining, and machine learning. Specifically, for example, querying group insurance policy data of a company. It should be understood that group insurance policies of a company include, but are not limited to, insurance policies of each employee in the company, insurance policies of items, and insurance policies of property, etc. Therefore, a lot of data is involved. In the process of data querying, Spark tasks can be created to realize the querying of such data.

[0033] In the specific implementation process, users create Spark tasks and determine a blacklist of processing task nodes at the Spark task initiator end. This allows the task receiver end to determine the blacklist of processing task nodes when it receives a Spark task, and then process the Spark task based on processing task nodes not on the blacklist, so that users can customize the configuration of processing task nodes.

[0034] Step S102: Determine multiple first task processing nodes required to process the Spark task based on the task processing information.

[0035] For example, after obtaining the task processing information of the Spark task, multiple first task processing nodes are determined based on the task processing information to process the Spark task. It should be understood that by distributing the Spark task to multiple first task processing nodes, the Spark task can be processed based on multiple computing resources, thereby providing computing power for big data computing and improving the processing efficiency of the Spark task.

[0036] It should be understood that, in addition to task processing nodes used to process Spark tasks, there are also task processing nodes for processing other tasks. These task processing nodes can be invoked to process different tasks. That is, if an idle task processing node is invoked to process a Spark task, then that task processing node is designated as the first task processing node. In the specific implementation process, if the first task processing node completes the processing of the Spark task and is in an idle state, the first task processing node is adjusted to process other tasks, so as to realize the invocation of the first task processing node to process other tasks. This application does not limit the specific invocation method of the task processing node.

[0037] Step S103: Verify each of the first task processing nodes based on the task processing node blacklist.

[0038] For example, it is determined whether the first task processing node is in the task processing node blacklist in order to select the task processing node.

[0039] In some embodiments, the method further includes: if the task processing node blacklist does not contain the first task processing node currently being verified, determining that the first task processing node currently being verified has passed verification; if the task processing node blacklist contains the first task processing node currently being verified, determining that the first task processing node currently being verified has failed verification.

[0040] For example, if the first task processing node is in the task processing blacklist, it is determined that the first task processing node has failed the verification; otherwise, the verification is passed, so that users can customize the task processing node used to process Spark tasks.

[0041] In some embodiments, the step of verifying each of the first task processing nodes based on the task processing node blacklist includes: verifying each of the first task processing nodes based on the driver process generated in the NodeManager of the Yarn cluster.

[0042] Please see Figure 2 , Figure 2 This is a scenario diagram illustrating a task processing method provided in an embodiment of this application. For example... Figure 2 As shown, for ease of understanding, the following embodiments use a Yarn cluster as an example.

[0043] For example, the first task processing node is verified based on the NodeManager in the Yarn cluster. Specifically, after receiving the task processing node blacklist and the first task processing node, the NodeManager generates a diver process to verify the first task processing node based on the task processing node blacklist, thus identifying the first task processing node that cannot execute Spark tasks. It should be understood that generating a diver process in the NodeManager to verify the first task processing node allows for direct filtering and removal of task processing nodes without needing to rebuild the Yarn cluster based on the task processing node blacklist, thereby improving the processing efficiency of Spark tasks.

[0044] In some embodiments, determining the multiple first task processing nodes required to process the Spark task based on the task processing information includes: a ResourceManager based on a Yarn cluster determining the multiple first task processing nodes required to process the Spark task based on the task information, and transmitting the first task processing nodes and the task processing node blacklist to the NodeManager.

[0045] For example, in a Yarn cluster, the ResourceManager receives task processing information from the Spark task initiator. Upon receiving this information, the ResourceManager determines the multiple first task processing nodes required to process the corresponding Spark task and sends these nodes, along with a blacklist of task processing nodes, to the NodeManager. The NodeManager then verifies the received first task processing nodes based on the blacklist. Specifically, the ResourceManager does not verify the first task processing nodes; it only determines the required resources based on the task processing information. The verification process is performed within the NodeManager. It should be understood that a Spark task comprises multiple task processes, which can be assigned to different first task processing nodes for processing, thereby improving task processing efficiency.

[0046] Step S104: Determine the first task processing node that has passed the verification as the second task processing node, and determine the task process corresponding to the second task processing node according to the task processing information.

[0047] In the specific implementation process, NodeManager will determine the first task processing node through verification as the second task processing node, and determine the task process corresponding to the second task processing node, so as to allocate the task process to the corresponding second task processing node. It should be understood that the second task processing node can also be a NodeManager.

[0048] In some embodiments, the method further includes: deleting the first task processing node that failed the verification based on the NodeManager, so as to use the remaining first task processing node as the second task processing node.

[0049] For example, in NodeManager, the first task processing node that fails the verification is deleted to prevent it from processing the corresponding task process, and the first task processing node that is not deleted is identified as the second task processing node. This achieves the filtering of task processing nodes to prevent the first task processing node in the task processing node blacklist from processing Spark tasks.

[0050] In the specific implementation process, after deleting the first task processing node from the task processing node blacklist and obtaining the second task processing node, NodeManager generates multiple task processes based on the task processing information and assigns the task processes to each of the second task processing nodes to realize task allocation and processing.

[0051] Step S105: Execute the task process based on the second task processing node.

[0052] For example, after receiving the assigned task process, the second task processing node processes the task process. It should be understood that when each second task processing node completes the processing of the assigned task process, the corresponding Spark task is also completed. This enables the use of multiple resources, such as multiple second task processing nodes, to process Spark tasks, thereby improving task processing efficiency. Furthermore, all second task processing nodes are task processing nodes outside the task processing blacklist, enabling user-defined task processing nodes.

[0053] It should be noted that in existing application scenarios of calling Yarn clusters, users often preset peak and off-peak periods to elastically schedule the computing resources of task processing nodes. For example, some task processing nodes are configured to expand into Yarn NodeManagers during peak daytime periods to allocate resources to T+1 type batch computing tasks. When these computing tasks are in off-peak periods at night, the NodeManager service on the task processing nodes that have been expanded into NodeManagers is removed so that the task processing nodes are returned to the OLAP service, thereby improving the operating efficiency of the cluster and reducing the cost of use. However, if the task processing node of the Yarn NodeManager is used to process Spark tasks, the existing architecture cannot guarantee that the task processing node can complete the corresponding task process or avoid being assigned a task process to the task processing node before removing the NodeManager service. As a result, when the task processing node needs to remove the NodeManager service, it may fail to be removed because it is still processing the corresponding task process, thus causing the service change to fail. This application solves the problem by setting a task processing node blacklist at the Spark task initiation end. For example, when a task processing node is preparing to change services, it is added to the task processing node blacklist so that the task processing node does not need to execute Spark tasks. This avoids the situation where the task processing node cannot change services because it is reassigned Spark tasks, thus enabling flexible adjustment of the cluster.

[0054] In some embodiments, the method further includes: when there is a task process that cannot be processed by the second task processing node, determining a process processing request based on the processing status of the task processing information; determining a third task processing node based on the process processing request, and verifying the third task processing node according to the task processing node blacklist; and if the third task processing node passes the verification, processing the task process that the second task processing node cannot process based on the third task processing node.

[0055] For example, in determining the task process, the process is determined based on the processes that the second task processing node can handle, so that the second task processing node can handle all task processes corresponding to the Spark task. However, in a specific implementation, both the task process and the first task processing node are determined in the ResourceManager. There is a corresponding relationship between the task process and the first task processing node. Since the ResourceManager does not verify the first task processing node, after verifying the first task processing node in the NodeManager, it will delete the first task processing node that is the same as the task processing node in the task processing node blacklist. This results in the number of second task processing nodes being less than the number of first task processing nodes, thus causing some task processes to not have a corresponding second task processing node. In this case, if the relationship between the task process and the first task processing node is one-to-many or many-to-many, after a certain first task processing node corresponding to the task process is deleted, it can still be determined whether there are other corresponding task processing nodes for the task process. If there are, and the task processing node is a second task processing node, then the task process is assigned to the corresponding second task processing node.

[0056] In other implementations, if the task process and task processing node are one-to-one and the first task processing node corresponding to the task process is deleted, or if the task process and first task processing node are one-to-many and all first task processing nodes corresponding to the task process are deleted, the NodeManager can determine the second task processing node that can handle these unrelated task processes from among multiple second task processing nodes, and assign these unrelated task processes to the capable second task nodes for processing; if the NodeManager determines that none of the current second task processing nodes can handle these task processes, it generates a process processing request based on the unhandled task processes and sends the process processing request to the ResourceManager, so that the ResourceManager can re-determine the task processing node according to the process processing request, so that the re-determined task processing node can handle these unhandled task processes.

[0057] In other implementations, NodeManager cannot determine whether the second task processing node can handle these task processes. That is, NodeManager does not have the function of determining the task processing node for handling task processes based on the task processes. Therefore, after NodeManager deletes the first task processing node so that the task processes do not have corresponding task processing nodes, it directly generates process processing requests based on these unrelated task processes and sends the process processing requests to ResourceManager so that ResourceManager can determine the task processing node for handling these task processes again based on the process processing requests.

[0058] Specifically, ResourceManager determines the third task processing node based on the process processing request and sends the third task processing node to NodeManager. Since ResourceManager does not verify the third task processing node, the third task processing node may still be in the task processing node blacklist. Therefore, after receiving the third task processing node, NodeManager verifies the third task processing node based on the task processing node blacklist. If the third task processing node passes the verification, the task process that was not assigned earlier is assigned to the third task processing node so that the third task processing node can handle the task process that the second task processing node failed to handle.

[0059] In some embodiments, the method further includes: if the third task processing node fails the verification, re-determining the task processing node based on the verification result of the third task processing node failing the verification and the process processing request.

[0060] In the specific implementation process, if the third task processing node fails the verification, the NodeManager will generate a process processing request again based on the verification result of the third task processing node failing the verification and the unprocessed task process, and send the process processing request to the ResourceManager so that the ResourceManager can determine the task processing node again based on the received process processing request. It should be understood that the re-determined task processing node does not include the previously determined third task processing node, so as to avoid repeated verification of the same task node.

[0061] The task processing method provided in the above embodiments, through a user-defined task processing node blacklist, enables users to customize task processing nodes in Spark task application scenarios. This makes Spark task resource scheduling more flexible, avoids assigning Spark task processes to task processing nodes that need to be changed, thus preventing errors in task processing nodes. Furthermore, when a second task processing node after verification fails to satisfy all task processes of a Spark task, a new task processing node can be determined, ensuring that the newly determined task processing node is not on the task processing node blacklist. This avoids errors when task processing nodes become services, and improves the flexibility of task processing node invocation and the processing efficiency of Spark tasks.

[0062] Please see Figure 3 , Figure 3 This is a schematic diagram of a task processing device provided in an embodiment of this application. The task processing device can be configured in a server or terminal to execute the aforementioned task processing method.

[0063] like Figure 3 As shown, the task processing device 100 includes: a task receiving module 110, a first node determination module 120, a node verification module 130, a second node determination module 140, and a task processing module 150.

[0064] The task receiving module 110 is used to receive task processing information of Spark tasks, including a blacklist of task processing nodes.

[0065] The first node determination module 120 is used to determine multiple first task processing nodes required to process the Spark task based on the task processing information.

[0066] The node verification module 130 is used to verify each of the first task processing nodes based on the task processing node blacklist.

[0067] The second node determination module 140 is used to determine the first task processing node that has passed the verification as the second task processing node, and to determine the task process corresponding to the second task processing node according to the task processing information.

[0068] The task processing module 150 is used to execute the task process based on the second task processing node.

[0069] For example, the node verification module 130 includes a NodeManager submodule, a text duplication rate calculation submodule, a second text determination submodule, a text deletion submodule, and a text matching submodule.

[0070] The NodeManager submodule is used to generate driver processes to verify each of the first task processing nodes based on the driver processes.

[0071] For example, the NodeManager submodule is further configured to determine that the first task processing node being verified has passed verification if the task processing node blacklist does not contain the first task processing node being verified; and to determine that the first task processing node being verified has failed verification if the task processing node blacklist contains the first task processing node being verified.

[0072] For example, the first node determination module 120 includes a ResourceManager submodule.

[0073] The ResourceManager submodule is used to determine multiple first task processing nodes required to process the Spark task based on the task information, and to transmit the first task processing nodes and the task processing node blacklist to the NodeManager submodule.

[0074] For example, the NodeManager submodule is also used to delete the first task processing node that fails the verification, so that the remaining first task processing node can be used as the second task processing node.

[0075] For example, the task processing device 100 also includes a request generation submodule.

[0076] The request generation submodule is used to determine the process processing request based on the processing status of the task processing information when there is a situation where the task process cannot be processed by the second task processing node.

[0077] The ResourceManager submodule is also used to determine the third task processing node based on the process processing request.

[0078] The NodeManager submodule is also used to verify the third task processing node based on the task processing node blacklist.

[0079] The task processing module 150 is also used to process task processes that the second task processing node cannot process, based on the third task processing node, if the third task processing node passes the verification.

[0080] For example, the NodeManager submodule is further configured to, if the third task processing node fails the verification, regenerate the process processing request based on the verification result of the third task processing node failing the verification and the process processing request, and send the regenerated process processing request to the ResourceManager submodule so that the ResourceManager submodule can re-determine the task processing node.

[0081] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] Please see Figure 4 , Figure 4 This is a schematic block diagram illustrating the structure of a computer device provided in an embodiment of this application. The computer device may be a server or a terminal.

[0083] like Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and internal memory.

[0084] The storage medium may store the operating system and computer programs. The computer programs include program instructions, which, when executed, cause the processor to perform any task processing method.

[0085] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0086] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When the computer program is executed by the processor, it enables the processor to perform any task processing method.

[0087] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0088] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0089] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0090] Receive task processing information for Spark tasks, including a blacklist of task processing nodes;

[0091] Based on the task processing information, determine the multiple first task processing nodes required to process the Spark task;

[0092] Verify each of the first task processing nodes based on the task processing node blacklist;

[0093] The first task processing node that passes the verification is determined as the second task processing node, and the task process corresponding to the second task processing node is determined according to the task processing information.

[0094] The task process is executed based on the second task processing node.

[0095] In one embodiment, when the processor performs verification on each of the first task processing nodes based on the task processing node blacklist, it is configured to:

[0096] The first task processing node is verified based on the driver process generated in the NodeManager of the Yarn cluster.

[0097] In one embodiment, the processor, when implementing the task processing method, is configured to:

[0098] If the task processing node blacklist does not contain the first task processing node currently being verified, then the first task processing node currently being verified is determined to have passed the verification.

[0099] If the task processing node blacklist contains the first task processing node currently being verified, then it is determined that the first task processing node currently being verified has failed the verification.

[0100] In one embodiment, when the processor determines, based on the task processing information, the plurality of first task processing nodes required to process the Spark task, it is configured to:

[0101] Based on the Yarn cluster, the ResourceManager determines multiple first task processing nodes required to process the Spark task according to the task information, and transmits the first task processing nodes and the task processing node blacklist to the NodeManager.

[0102] In one embodiment, the processor, when implementing the task processing method, is configured to:

[0103] Based on the NodeManager, the first task processing node that fails the verification is deleted, and the remaining first task processing node is used as the second task processing node.

[0104] In one embodiment, the processor, when implementing the task processing method, is configured to:

[0105] In the event that a task process cannot be processed by the second task processing node, the process processing request is determined based on the processing status of the task processing information.

[0106] The third task processing node is determined based on the process processing request, and the third task processing node is verified according to the task processing node blacklist.

[0107] If the third task processing node passes the verification, the task process that the second task processing node cannot process is processed based on the third task processing node.

[0108] In one embodiment, the processor, when implementing the task processing method, is configured to:

[0109] If the third task processing node fails the verification, the task processing node is re-determined based on the verification result of the third task processing node failing the verification and the process processing request.

[0110] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the computer device described above can be referred to the corresponding process in the aforementioned task processing method embodiments, and will not be repeated here.

[0111] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the task processing method of this application.

[0112] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0113] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0114] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0115] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task processing method characterized by, The method comprises the following steps: receiving task processing information of a Spark task, the task processing information comprising a task processing node blacklist; determining a plurality of first task processing nodes required for processing the Spark task according to the task processing information, comprising: determining a plurality of first task processing nodes required for processing the Spark task according to the task processing information based on a ResourceManager of a Yarn cluster, and transmitting the first task processing nodes and the task processing node blacklist to a NodeManager of the Yarn cluster; checking each of the first task processing nodes based on the task processing node blacklist, comprising: based on a driver process generated in the NodeManager, determining that a currently checked first task processing node passes the check when the task processing node blacklist does not contain the currently checked first task processing node; and determining that the currently checked first task processing node fails the check when the task processing node blacklist contains the currently checked first task processing node; determining the first task processing nodes that pass the check as second task processing nodes, comprising: based on the NodeManager, deleting the first task processing nodes that fail the check to take the remaining first task processing nodes as the second task processing nodes; determining a task process corresponding to the second task processing nodes according to the task processing information; executing the task process based on the second task processing nodes; in the case where the task process cannot be processed by the second task processing nodes, determining third task processing nodes according to the processing condition of the task processing information, and checking the third task processing nodes according to the task processing node blacklist; in the case where the third task processing nodes pass the check, processing the task process that cannot be processed by the second task processing nodes based on the third task processing nodes. The method further comprises: setting a task processing node blacklist at a Spark task initiation end, and writing a task processing node that is ready to change service into the task processing node blacklist.

2. The task processing method of claim 1, wherein, The method further comprises: if the third task processing nodes fail the check, re-determining task processing nodes according to the check result of the third task processing nodes failing the check and the process processing request.

3. A task processing apparatus characterized by comprising: The task processing device comprises: a task receiving module configured to receive task processing information of a Spark task, the task processing information comprising a task processing node blacklist; The first node determination module is configured to determine, according to the task processing information, a plurality of first task processing nodes required for processing the Spark task; the first node determination module includes a ResourceManager submodule configured to determine, according to the task processing information, a plurality of first task processing nodes required for processing the Spark task, and transmit the first task processing nodes and the task processing node blacklist to a NodeManager submodule of a Yarn cluster; The node checking module is configured to check each of the first task processing nodes based on the task processing node blacklist; the node checking module includes a NodeManager submodule configured to, based on the generated driver process, determine that a currently checked first task processing node passes the check when the task processing node blacklist does not contain the currently checked first task processing node, and determine that the currently checked first task processing node fails the check when the task processing node blacklist contains the currently checked first task processing node; The second node determination module is configured to determine the first task processing node that passes the check as a second task processing node; the NodeManager submodule is further configured to delete the first task processing node that fails the check, so as to take the remaining first task processing nodes as the second task processing nodes; The second node determination module is further configured to determine, according to the task processing information, a task process corresponding to the second task processing node; The task processing module is configured to execute the task process based on the second task processing node; The ResourceManager submodule is further configured to determine, according to the processing condition of the task processing information, a third task processing node when the request generation submodule determines that there is a condition that the task process cannot be processed by the second task processing node; and the task processing module is further configured to process the task process that cannot be processed by the second task processing node based on the third task processing node when the third task processing node passes the check. The task processing device is further configured to set a task processing node blacklist at a Spark task initiation end, and write a task processing node that is ready to change service into the task processing node blacklist.

4. A computer device, comprising: The computer device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the task processing method according to claim 1 or 2.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the task processing method according to claim 1 or 2.

Images