Task scheduling method, device and equipment applied to spark system

Abstract

The invention provides a task scheduling method, device and equipment applied to a spark system. The method comprises the following steps: receiving a total task of a spark system sent by a client, wherein the total task comprises a preset number of subtasks; calculating to obtain an initial resource allocation parameter according to the data volume of the preset number of subtasks and the available resource amount of each calculation node; applying for corresponding initial resources from a resource management system according to the initial resource allocation parameters; classifying the preset number of subtasks into a plurality of stage tasks; according to the parent-child dependency relationship model, scheduling the stage task with the parent-child dependency relationship to one computing node in the initial resource for task processing; and for all the sub-tasks of any computing node, scheduling the sub-tasks with communication association to the same resource container in any computing node according to the communication association model for task processing. Computing resources can be fully utilized and saved, and the cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of cloud computing resource scheduling, and in particular to a task scheduling method, device and equipment applied to a spark system. Background Art

[0002] In recent years, with the rapid development of technologies such as social networks and the Internet of Things, the demand for large-scale data analysis and the growth of massive data have put forward new requirements for storage management and computing analysis. Big data processing has become crucial. The Spark (Apache Spark, big data analysis engine) framework has been widely used in big data processing, and how to improve Spark performance has become a hot topic.

[0003] In the prior art, Spark performance is mainly related to task scheduling, and there are two default task scheduling strategies: FIFO mode and Fair sharing mode. In FIFO mode, Spark considers mobile computing by default and tries to allocate tasks in the Stage to the nodes that store the input data required by the task.

[0004] However, the task scheduling method in the prior art does not consider the correlation between tasks, which may cause some nodes in the cluster to be overloaded while some nodes are in an "idle" state, seriously wasting the computing resources of the cluster. Summary of the invention

[0005] The present application provides a task scheduling method, device and equipment applied to a spark system, so as to solve the technical problem that the task scheduling method in the prior art only considers the correlation at the stage task level, which may cause cluster load imbalance and waste computing resources.

[0006] In a first aspect, the present application provides a task scheduling method applied to a spark system, comprising:

[0007] Receive a total task of a big data analysis engine spark system sent by a client, wherein the total task includes a preset number of subtasks;

[0008] Obtaining the available amount of resources of each computing node in the resource management system and the amount of data of the preset number of subtasks;

[0009] Calculating initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available amount of resources of each computing node;

[0010] Applying for corresponding initial resources from the resource management system according to the initial resource allocation parameters, wherein the initial resources belong to a plurality of computing nodes;

[0011] Classifying the preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks;

[0012] Establish a parent-child dependency model between tasks at each stage;

[0013] According to the parent-child dependency model, scheduling the stage tasks with the parent-child dependency to a computing node in the initial resources for task processing;

[0014] For all subtasks of any computing node, a communication association model between subtasks is established;

[0015] Subtasks with communication association are scheduled to the same resource container in any computing node for task processing according to the communication association model.

[0016] Optionally, the method as described above, wherein the initial resource allocation parameters for the preset number of subtasks are calculated based on the data volume of the preset number of subtasks and the available resources of each computing node, includes: determining the resource requirement based on the data volume of the preset number of subtasks, wherein the resource requirement is the minimum amount of resources for processing the data volume of the preset number of subtasks; and determining the initial resource allocation parameters from the available resources of each computing node according to a preset multiple based on the resource requirement.

[0017] Optionally, in the method as described above, the resource requirement includes: total node requirement, total memory requirement and total processor requirement; accordingly, the initial resource allocation parameters are determined from the available resources of each computing node according to preset multiples based on the resource requirements, including: determining the number of computing nodes in the initial resource allocation parameters from each computing node according to a first preset multiple of the total node requirement; determining the total memory size in the initial resource allocation parameters from each computing node according to a second preset multiple of the total memory requirement; and determining the total number of processors in the initial resource allocation parameters from each computing node according to a third preset multiple of the total processor requirement.

[0018] Optionally, as described above, the method classifies the preset number of subtasks into multiple stage tasks, wherein each stage task includes multiple subtasks, including: obtaining an operator of the total task of the spark system; and dividing the total task of the spark system into multiple stage tasks according to the operator.

[0019] Optionally, the method as described above, after scheduling the subtasks with communication association to the same resource container in any one of the computing nodes for task processing according to the communication association model, further includes: obtaining the data volume of the remaining subtasks and the remaining available resources of each computing node of the resource management system at preset time intervals; calculating resource allocation parameters for the remaining subtasks according to the data volume of the remaining subtasks and the remaining available resources of each computing node; and applying for corresponding resources from the resource management system according to the resource allocation parameters, wherein the resources belong to multiple computing nodes.

[0020] Optionally, the method as described above, wherein the resource allocation parameters for the remaining subtasks are calculated based on the data volume of the remaining subtasks and the available resources of each computing node, includes: determining the remaining resource requirement based on the data volume of the remaining subtasks, wherein the remaining resource requirement is the minimum amount of resources for processing the data volume of the remaining subtasks; and determining the resource allocation parameters from the remaining available resources of each computing node according to a preset multiple based on the remaining resource requirement.

[0021] Optionally, in the method as described above, the remaining resource requirement includes: remaining memory requirement and remaining processor requirement; accordingly, determining the resource allocation parameters from the remaining resource available amount of each computing node according to preset multiples based on the remaining resource requirement includes: determining the remaining memory size in the resource allocation parameters from each computing node according to a second preset multiple of the remaining memory requirement; and determining the remaining number of processors in the resource allocation parameters from each computing node according to a third preset multiple of the remaining processor requirement.

[0022] In a second aspect, the present application provides a task scheduling device applied to a spark system, comprising:

[0023] A receiving module is used to receive a total task of the spark system sent by a client, wherein the total task includes a preset number of subtasks;

[0024] An acquisition module, used to acquire the available amount of resources of each computing node in the resource management system and the amount of data of the preset number of subtasks;

[0025] A parameter calculation module, used to calculate initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available resources of each computing node;

[0026] A resource application module, used to apply for corresponding initial resources from the resource management system according to the initial resource allocation parameters, wherein the initial resources belong to multiple computing nodes;

[0027] A task classification module, used for classifying the preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks;

[0028] The first building module is used to establish the parent-child dependency model between tasks in each stage;

[0029] A first scheduling module, configured to schedule the stage tasks with parent-child dependency relationships to a computing node in the initial resources for task processing according to the parent-child dependency model;

[0030] The second building module is used to establish a communication association model between all subtasks of any computing node;

[0031] The second scheduling module is used to schedule the subtasks with communication association to the same resource container in any computing node for task processing according to the communication association model.

[0032] In a third aspect, the present application provides a service device, comprising: a processor, and a memory communicatively connected to the processor;

[0033] The memory stores computer-executable instructions;

[0034] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the task scheduling method applied to the spark system as described in the first aspect and various possible designs of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect and various possible designs of the first aspect.

[0036] The task scheduling method, device and equipment applied to the spark system provided by the present application establish a parent-child dependency model between tasks at each stage, schedule the stage tasks with parent-child dependency to a computing node in the initial resources for task processing, establish a communication association model between subtasks in any computing node, schedule subtasks with communication association to the same resource container in any of the above computing nodes for task processing, optimize task scheduling from the stage task and subtask levels, centrally process related tasks, make full use of computing resources, save computing resources and reduce costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] Figure 1 A schematic diagram of a scenario of a task scheduling method applied to a spark system provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of a task scheduling method applied to a spark system provided by an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of a task scheduling device applied to a spark system provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the hardware structure of the service device provided in an embodiment of the present application.

[0042] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 1A system diagram of a task scheduling method applied to a spark system provided in an embodiment of the present application. Figure 1 As shown, the system provided by this embodiment includes a client 101 , a service device 102 and a resource management system 103 .

[0047] Specifically, the service device 102 receives the total tasks of the spark system sent by the client 101, calculates resource allocation parameters for the tasks of the spark system, and applies for resources from the resource management system 103 based on the resource allocation parameters.

[0048] Figure 2 A flowchart of a task scheduling method for a spark system provided by an embodiment of the present application. The execution subject of this embodiment can be Figure 1 The service device 102 shown in FIG. 1 may also be other computer devices, which is not particularly limited in this embodiment. Figure 2 As shown, the method includes:

[0049] S201: Receive a total task of a big data analysis engine spark system sent by a client, wherein the total task includes a preset number of subtasks.

[0050] Among them, each subtask can be one or more of a data calculation task, an image processing task, and a model training task.

[0051] S202: Obtain the available amount of resources of each computing node in the resource management system and the amount of data of a preset number of subtasks.

[0052] The available resources of each computing node include available nodes, available memory, and available processors.

[0053] S203: Calculate and obtain initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available resources of each computing node.

[0054] Specifically, S203 includes S2031 to S2032:

[0055] S2031: Determine resource requirements according to a preset number of subtask data volumes, wherein the resource requirements are minimum resource requirements for processing the preset number of subtask data volumes.

[0056] The resource requirements include: total node requirements, total memory requirements, and total processor requirements.

[0057] S2032: Determine initial resource allocation parameters from available resources of each computing node according to the resource demand and a preset multiple.

[0058] Specifically, S2032 specifically includes Sa~Sc:

[0059] Sa: Determine the number of computing nodes in the initial resource allocation parameters from each computing node according to a first preset multiple of the total node demand.

[0060] Exemplarily, the total node requirement is a, the first preset multiple is b, and the number of computing nodes in the initial resource allocation parameters is a*b.

[0061] Sb: Determine the total memory size in the initial resource allocation parameters from each computing node according to a second preset multiple of the total memory requirement.

[0062] Exemplarily, the total memory requirement is c, the second preset multiple is d, and the total memory size in the initial resource allocation parameter is c*d.

[0063] Sc: Determine the total number of processors in the initial resource allocation parameters from each computing node according to a third preset multiple of the total processor requirement.

[0064] Exemplarily, the total processor requirement is e, the third preset multiple is f, and the total number of processors in the initial resource allocation parameters is e*f.

[0065] S204: Apply for corresponding initial resources from the resource management system according to the initial resource allocation parameters, where the initial resources belong to multiple computing nodes.

[0066] Specifically, an initial resource application form is generated according to the initial resource allocation parameters, and the initial resource application form is sent to the resource management system to apply for initial resources.

[0067] S205: Classify a preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks.

[0068] Specifically, S205 specifically includes Sd to Se:

[0069] Sd: Gets the operator of the total tasks of the spark system.

[0070] Specifically, the string with specific characters in the total task of the spark system is parsed to obtain the operator.

[0071] Se: Divide the total task of the Spark system into multiple stage tasks according to the operator.

[0072] Specifically, the action operator and the wide dependency operator in the operator are obtained, the total task of the spark system is divided into multiple engineering tasks according to the action operator, and the engineering task is divided into multiple stage tasks according to the wide dependency operator.

[0073] Among them, each engineering task contains an action operator.

[0074] S206: Establish a parent-child dependency model between tasks in each stage.

[0075] The parent-child dependency model can be of table type or layout type.

[0076] S207: According to the parent-child dependency model, the stage tasks with the parent-child dependency are scheduled to a computing node in the initial resources for task processing.

[0077] Specifically, the stage tasks with parent-child dependency are acquired according to the parent-child dependency model, and the stage tasks with parent-child dependency are scheduled to a computing node in the initial resources for task processing.

[0078] S208: For all subtasks of any computing node, a communication association model between the subtasks is established.

[0079] The communication association model may be of a table type or a layout type.

[0080] S209: Scheduling the subtasks with communication association to the same resource container in any computing node for task processing according to the communication association model.

[0081] Specifically, subtasks with communication association are acquired according to the communication association model, and the subtasks with communication association are scheduled to the same resource container in any computing node for task processing.

[0082] From the above description, it can be seen that the present application establishes a parent-child dependency model between tasks at each stage, and schedules the stage tasks with parent-child dependencies to a computing node in the initial resources for task processing. By establishing a communication association model between subtasks in any computing node, the subtasks with communication associations are scheduled to the same resource container in any of the above computing nodes for task processing. Task scheduling is optimized at the stage task and subtask levels, and related tasks are processed centrally to make full use of computing resources, save computing resources, and reduce costs.

[0083] In one embodiment of the present application, based on the above embodiment, after step S209, a process of adaptively adjusting the resource allocation parameters is further included, which is described in detail as follows:

[0084] S301: Receive a total task of a big data analysis engine spark system sent by a client, wherein the total task includes a preset number of subtasks.

[0085] S302: Obtain the available amount of resources of each computing node in the resource management system and the amount of data of a preset number of subtasks.

[0086] S303: Calculate and obtain initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available resources of each computing node.

[0087] S304: Apply for corresponding initial resources from the resource management system according to the initial resource allocation parameters, where the initial resources belong to multiple computing nodes.

[0088] S305: Classify a preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks.

[0089] S306: Establish a parent-child dependency model between tasks in each stage.

[0090] S307: According to the parent-child dependency model, the stage task with the parent-child dependency relationship is scheduled to a computing node in the initial resources for task processing.

[0091] S308: For all subtasks of any computing node, a communication association model between the subtasks is established.

[0092] S309: Scheduling the subtasks with communication association to the same resource container in any computing node for task processing according to the communication association model.

[0093] In this embodiment, the description of steps S301 to S309 is consistent with the above-mentioned steps S201 to S209. For specific descriptions, please refer to the relevant contents of steps S201 to S209, which will not be repeated here.

[0094] S310: Obtain the data volume of the remaining subtasks and the remaining available resources of each computing node of the resource management system at every preset time period.

[0095] The remaining available resources of each computing node include the remaining available memory and the remaining available processor.

[0096] S311: Calculate resource allocation parameters for the remaining subtasks according to the data volume of the remaining subtasks and the remaining available resources of each computing node.

[0097] Specifically, S311 includes S3111 to S3112:

[0098] S3111: Determine the remaining resource requirement based on the data volume of the remaining subtasks, where the remaining resource requirement is the minimum resource amount for processing the data volume of the remaining subtasks.

[0099] The remaining resource requirements include remaining memory requirements and remaining processor requirements.

[0100] S3112: Determine resource allocation parameters from the remaining available resources of each computing node according to the preset multiple based on the remaining resource demand.

[0101] Specifically, S3112 specifically includes Sf~Sg:

[0102] Sf: Determine the remaining memory size in the resource allocation parameter from each computing node according to the second preset multiple of the remaining memory requirement.

[0103] Exemplarily, the remaining memory requirement is g, the second preset multiple is d, and the remaining memory size in the resource allocation parameter is g*d.

[0104] Sg: Determine the number of remaining processors in the resource allocation parameters from each computing node according to a third preset multiple of the remaining processor demand.

[0105] Exemplarily, the remaining processor requirement is h, the third preset multiple is f, and the number of remaining processors in the resource allocation parameter is h*f.

[0106] S312: Apply for corresponding resources from the resource management system according to the resource allocation parameters, where the resources belong to multiple computing nodes.

[0107] Specifically, a resource application form is generated according to resource allocation parameters, and the resource application form is sent to a resource management system to apply for resources.

[0108] From the above description, it can be seen that the present application calculates the resource allocation parameters for the remaining subtasks according to the data volume of the remaining subtasks and the remaining available resources of each computing node at preset time intervals. It does not need to use training models and large data sets for large-scale training, and can perform adaptive parameter recommendations. It does not need to spend a lot of time on manual adjustments, and the operation is simple, saving time costs.

[0109] Figure 3 The structure diagram of the task scheduling device applied to the spark system provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the task scheduling device 30 applied to the spark system includes: a receiving module 301, an acquisition module 302, a parameter calculation module 303, a resource application module 304, a task classification module 305, a first construction module 306, a first scheduling module 307, a second construction module 308 and a second scheduling module 309.

[0110] The receiving module 301 is used to receive a total task of the spark system sent by the client, wherein the total task includes a preset number of subtasks;

[0111] An acquisition module 302 is used to acquire the available amount of resources of each computing node in the resource management system and the amount of data of the preset number of subtasks;

[0112] A parameter calculation module 303, configured to calculate initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available resources of each computing node;

[0113] A resource application module 304, configured to apply for corresponding initial resources from the resource management system according to the initial resource allocation parameters, wherein the initial resources belong to a plurality of computing nodes;

[0114] A task classification module 305 is used to classify the preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks;

[0115] The first construction module 306 is used to establish a parent-child dependency relationship model between tasks in each stage;

[0116] A first scheduling module 307 is used to schedule the stage tasks with parent-child dependency to a computing node in the initial resources for task processing according to the parent-child dependency model;

[0117] The second building module 308 is used to establish a communication association model between all subtasks of any computing node;

[0118] The second scheduling module 309 is used to schedule the subtasks with communication association to the same resource container in any computing node for task processing according to the communication association model.

[0119] In a possible design, the parameter calculation module 303 includes:

[0120] A first determining unit 3031 is used to determine a resource requirement according to the data volume of the preset number of subtasks, wherein the resource requirement is a minimum resource amount for processing the data volume of the preset number of subtasks;

[0121] The second determining unit 3032 is used to determine the initial resource allocation parameters from the available resources of each computing node according to the resource demand and a preset multiple.

[0122] In one possible design, the resource requirement includes: total node requirement, total memory requirement and total processor requirement; the second determination unit 3032 is specifically used to: determine the number of computing nodes in the initial resource allocation parameters from the computing nodes according to a first preset multiple of the total node requirement; determine the total memory size in the initial resource allocation parameters from the computing nodes according to a second preset multiple of the total memory requirement; determine the total number of processors in the initial resource allocation parameters from the computing nodes according to a third preset multiple of the total processor requirement.

[0123] In a possible design, the task classification module 305 is specifically used to: obtain the operator of the total task of the spark system; and divide the total task of the spark system into multiple stage tasks according to the operator.

[0124] In one possible design, the acquisition module 302 is also used to obtain the data volume of the remaining subtasks and the remaining available resources of each computing node of the resource management system at preset time intervals; the parameter calculation module 303 is also used to calculate the resource allocation parameters for the remaining subtasks based on the data volume of the remaining subtasks and the remaining available resources of each computing node; the resource application module 304 is also used to apply for corresponding resources from the resource management system based on the resource allocation parameters, wherein the resources belong to multiple computing nodes.

[0125] In a possible design, the parameter calculation module 303 further includes:

[0126] A third determining unit 3033 is used to determine the remaining resource requirement according to the data volume of the remaining subtasks, wherein the remaining resource requirement is the minimum resource amount for processing the data volume of the remaining subtasks;

[0127] The fourth determining unit 3034 is used to determine the resource allocation parameter from the remaining resource available amount of each computing node according to the remaining resource demand and a preset multiple.

[0128] In one possible design, the remaining resource requirement includes: remaining memory requirement and remaining processor requirement; the fourth determination unit 3034 is specifically used to: determine the remaining memory size in the resource allocation parameter from each computing node according to the second preset multiple of the remaining memory requirement; determine the remaining number of processors in the resource allocation parameter from each computing node according to the third preset multiple of the remaining processor requirement.

[0129] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0130] Figure 4 Schematic diagram of the hardware structure of the service device provided in the embodiment of the present application. Figure 4 As shown, the service device 40 of this embodiment includes: at least one processor 401 and a memory 402; the memory stores computer execution instructions; at least one processor executes the computer execution instructions stored in the memory, so that at least one processor executes the task scheduling method applied to the spark system as above;

[0131] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0132] When the memory 402 is independently provided, the service device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0133] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the task scheduling method applied to the spark system as described above is implemented.

[0134] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the task scheduling method applied to the spark system as described above.

[0135] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0136] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0137] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0138] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0139] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0140] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0143] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A task scheduling method applied to a spark system, characterized in that: Applied to service equipment, including: Receive a total task of a big data analysis engine spark system sent by a client, wherein the total task includes a preset number of subtasks; Obtaining the available amount of resources of each computing node in the resource management system and the amount of data of the preset number of subtasks; Calculating initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available amount of resources of each computing node; Applying for corresponding initial resources from the resource management system according to the initial resource allocation parameters, wherein the initial resources belong to a plurality of computing nodes; Classifying the preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks; Establish a parent-child dependency model between tasks at each stage; According to the parent-child dependency model, scheduling the stage tasks with the parent-child dependency to a computing node in the initial resources for task processing; For all subtasks of any computing node, a communication association model between subtasks is established; Subtasks with communication association are scheduled to the same resource container in any computing node for task processing according to the communication association model.

2. The method according to claim 1, characterized in that: The calculating, according to the data amount of the preset number of subtasks and the available amount of resources of each computing node, initial resource allocation parameters for the preset number of subtasks include: Determining a resource requirement according to the data volume of the preset number of subtasks, wherein the resource requirement is a minimum resource amount for processing the data volume of the preset number of subtasks; According to the resource demand, initial resource allocation parameters are determined from the available resources of each computing node according to a preset multiple.

3. The method according to claim 2, characterized in that The resource requirements include: total node requirements, total memory requirements and total processor requirements; Accordingly, determining the initial resource allocation parameters from the available resources of each computing node according to the resource demand and a preset multiple includes: Determining the number of computing nodes in the initial resource allocation parameter from the computing nodes according to a first preset multiple of the total node demand; Determining a total memory size in an initial resource allocation parameter from each computing node according to a second preset multiple of the total memory requirement; According to the third preset multiple of the total processor requirement, the total number of processors in the initial resource allocation parameters is determined from the computing nodes.

4. The method according to claim 1, characterized in that: The method of classifying the preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks, includes: Get the operator of the total task of the spark system; The total task of the spark system is divided into multiple stage tasks according to the operator.

5. The method according to any one of claims 1 to 4, characterized in that: After scheduling the subtasks with communication association to the same resource container in any computing node for task processing according to the communication association model, the method further includes: Obtaining the data volume of the remaining subtasks and the remaining available resources of each computing node of the resource management system at every preset time period; Calculating resource allocation parameters for the remaining subtasks according to the data volume of the remaining subtasks and the remaining available resources of each computing node; According to the resource allocation parameters, corresponding resources are applied to the resource management system, wherein the resources belong to a plurality of computing nodes.

6. The method according to claim 5, characterized in that The calculating, according to the data amount of the remaining subtasks and the available amount of resources of each computing node, resource allocation parameters for the remaining subtasks comprises: Determine the remaining resource requirement according to the data volume of the remaining subtasks, wherein the remaining resource requirement is the minimum resource amount for processing the data volume of the remaining subtasks; According to the remaining resource demand, resource allocation parameters are determined from the remaining available resources of each computing node according to a preset multiple.

7. The method according to claim 6, characterized in that The remaining resource requirements include: remaining memory requirements and remaining processor requirements; Accordingly, determining the resource allocation parameter from the remaining available resources of each computing node according to the remaining resource demand and a preset multiple includes: Determining the remaining memory size in the resource allocation parameter from each computing node according to a second preset multiple of the remaining memory requirement; According to the third preset multiple of the remaining processor requirement, the number of remaining processors in the resource allocation parameter is determined from each computing node.

8. A task scheduling device applied to a spark system, characterized in that: Applied to service equipment, including: A receiving module is used to receive a total task of the spark system sent by a client, wherein the total task includes a preset number of subtasks; An acquisition module, used to acquire the available amount of resources of each computing node in the resource management system and the amount of data of the preset number of subtasks; A parameter calculation module, used to calculate initial resource allocation parameters for the preset number of subtasks according to the data volume of the preset number of subtasks and the available resources of each computing node; A resource application module, used to apply for corresponding initial resources from the resource management system according to the initial resource allocation parameters, wherein the initial resources belong to multiple computing nodes; A task classification module, used for classifying the preset number of subtasks into a plurality of stage tasks, wherein each stage task includes a plurality of subtasks; The first building module is used to establish the parent-child dependency model between tasks in each stage; A first scheduling module, configured to schedule the stage tasks with parent-child dependency relationships to a computing node in the initial resources for task processing according to the parent-child dependency model; The second building module is used to establish a communication association model between all subtasks of any computing node; The second scheduling module is used to schedule the subtasks with communication association to the same resource container in any computing node for task processing according to the communication association model.

9. A service device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the task scheduling method applied to the spark system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the task scheduling method applied to a spark system as described in any one of claims 1 to 7.

Images