Hybrid cloud deployment cluster Spark job scheduling method, apparatus and device, and storage medium

Abstract

The invention discloses a hybrid cloud deployment cluster Spark job scheduling method and device, equipment and a storage medium. The method comprises the following steps: acquiring current operation information, virtual machine information and stage information in real time; on the basis of the operation information, the virtual machine information and the stage information, adopting a particle swarm optimization algorithm, and taking minimization of the virtual machine use cost of the whole cluster as an optimization target to generate current scheduling information; and scheduling an executor for execution according to the scheduling information. According to the technical scheme provided by the invention, the job information, the virtual machine information and the stage information are combined, and the optimal scheduling scheme is accurately customized for the DAG of each Spark job in a multi-job environment through the iterative search of the particle swarm based on the meta-heuristic particle swarm optimization algorithm, so that the overall operation cost of a hybrid cloud deployment cluster is effectively reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data processing technology, and in particular to a hybrid cloud deployment cluster Spark job scheduling method, device, equipment and storage medium. Background Art

[0002] Some people prefer to deploy big data clusters in local environments, while others prefer to build big data clusters in the cloud because cloud service providers can provide flexible, economical, and highly scalable computing resources. In addition to a single deployment model, the current trend is that more and more professionals are beginning to explore hybrid deployment strategies that combine local and cloud resources to build more efficient clusters. Spark effectively bypasses expensive disk and network I / O operations through its in-memory Resilient Distributed Datasets (RDD) technology, significantly improving data processing efficiency. Given the strong data dependencies inherent in many data analysis tasks, this dependency is often efficiently represented and managed through directed acyclic graphs (DAGs). Spark has built-in scheduling algorithms including FIFO and Fair for resource allocation and task execution.

[0003] Currently, the mainstream method for scheduling Spark jobs in hybrid cloud deployment clusters is mainly the Greedy Iterative Optimization (GIO) algorithm, which uses the pricing model of different virtual machine (VM) instances and the estimated completion time of each job to find cost-effective executor placement (based on each executor). The method focusing on DAG scheduling of Spark jobs is mainly Decima, which uses reinforcement learning (RL) and neural networks to learn the DAG form of different Spark jobs, develop new DAG representations, and handle the complexity of scheduling problems to complete scheduling.

[0004] The GIO algorithm focuses on hybrid cloud deployment of Spark clusters and cost optimization, but does not pay attention to the DAG details of each Spark job, nor does it consider the DAG scheduling of each Spark job. Although Decima pays attention to the DAG form of different Spark jobs, it also considers the complexity of scheduling problems and performs scheduling based on the DAG of Spark jobs, but it does not consider the complexity of hybrid cloud clusters and cost optimization issues. Therefore, existing Spark job scheduling algorithms have limitations. They either only focus on high-level scheduling, ignore the DAG relationship within the job, and cannot achieve fine-grained optimization, or they do not fully utilize the heterogeneous characteristics of hybrid cloud clusters and do not consider cost optimization enough. Summary of the invention

[0005] The embodiments of the present invention provide a hybrid cloud deployment cluster Spark job scheduling method, apparatus, device and storage medium to select the optimal scheduling solution for the DAG of each Spark job in a multi-job environment, thereby minimizing the cost of using virtual machines.

[0006] In a first aspect, an embodiment of the present invention provides a method for scheduling Spark jobs in a hybrid cloud deployment cluster, the method comprising:

[0007] Get current job information, virtual machine information, and stage information in real time;

[0008] Based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as an optimization goal to generate current scheduling information;

[0009] The executor is scheduled to execute according to the scheduling information.

[0010] Optionally, based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as an optimization goal to generate the current scheduling information, including:

[0011] Create a particle swarm and randomly initialize the position and velocity of each particle. Each particle represents a possible placement scheme in the hybrid cloud deployment cluster during the job phase.

[0012] Dividing the particle swarm into a plurality of groups, and determining an optimal placement scheme in each group with the lowest virtual machine usage cost;

[0013] updating the position and velocity of each particle in the corresponding group according to the optimal placement scheme in the group, and generating a new placement scheme for the corresponding particle based on the updated position and velocity;

[0014] Iteratively re-determine the optimal placement plan within each group based on the new placement plan until a preset termination condition is reached, determine the global optimal placement plan based on the final optimal placement plan within the group, and generate the scheduling information based on the global optimal placement plan.

[0015] Optionally, after updating the position and speed of each particle in the corresponding group according to the optimal placement plan in the group and before generating a new placement plan for the corresponding particle based on the updated position and speed, the method further includes:

[0016] A multi-point crossover strategy is used to cross-update the particles in each group.

[0017] Optionally, the optimization objective is specifically:

[0018] min:Cost total =Cost L +Cost C ;

[0019]

[0020] Among them, Cost total Indicates the virtual machine usage cost of the entire cluster. L Indicates the cost of using the local virtual machine. C represents the cost of using cloud virtual machines, M represents the total number of local virtual machines, and x j Indicates the usage status of the jth local virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the jth local virtual machine, T C represents the estimated completion time of the current job, N represents the total number of cloud virtual machines, and y k Indicates the usage status of the kth cloud virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the k-th cloud virtual machine, Indicates the estimated completion time of the current job when cloud virtual machines are used.

[0021] Optionally, the optimization objective is based on the following constraints:

[0022]

[0023] Where E represents the total number of stages required for the current job, u ij Indicates whether the i-th stage is placed in the j-th local virtual machine, 1 if yes, 0 if no. represents the CPU resources required for the i-th stage, represents the currently available CPU resources in the jth local virtual machine, v ik Indicates whether the i-th stage is placed in the k-th cloud virtual machine, 1 if yes, 0 if no. Indicates the CPU resources currently available in the k-th cloud virtual machine.

[0024] Optionally, the scheduling information includes: a stage scheduling order provided to a directed acyclic graph scheduler, and a correspondence between tasks and executors provided to a task scheduler.

[0025] In a second aspect, an embodiment of the present invention further provides a hybrid cloud deployment cluster Spark job scheduling device, the device comprising:

[0026] Information acquisition module, used to obtain current job information, virtual machine information and stage information in real time;

[0027] A scheduling information generation module, configured to generate current scheduling information based on the job information, the virtual machine information and the stage information by using a particle swarm optimization algorithm with the optimization goal of minimizing the virtual machine usage cost of the entire cluster;

[0028] The task execution module is used to schedule the executor to execute according to the scheduling information.

[0029] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0030] one or more processors;

[0031] A memory for storing one or more programs;

[0032] When the one or more programs are executed by the one or more processors, the one or more processors implement the hybrid cloud deployment cluster Spark job scheduling method provided in any embodiment of the present invention.

[0033] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the hybrid cloud deployment cluster Spark job scheduling method provided by any embodiment of the present invention is implemented.

[0034] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program, and when the program is executed by a processor, it implements the hybrid cloud deployment cluster Spark job scheduling method provided by any embodiment of the present invention.

[0035] The embodiment of the present invention provides a method for scheduling Spark jobs in a hybrid cloud deployment cluster. First, the current job information, virtual machine information and stage information are obtained in real time. Then, based on this information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as the optimization goal, and the current scheduling information is generated, so as to schedule the executor to execute according to the obtained scheduling information. The method for scheduling Spark jobs in a hybrid cloud deployment cluster provided by the embodiment of the present invention combines job information, virtual machine information and stage information, and is based on a meta-heuristic particle swarm optimization algorithm. Through iterative search of a particle swarm, it is possible to accurately customize the optimal scheduling solution for the DAG of each Spark job in a multi-job environment, thereby effectively reducing the overall operating cost of the hybrid cloud deployment cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a hybrid cloud deployment cluster Spark job scheduling method provided in Example 1 of the present invention;

[0037] Figure 2A schematic diagram of a scheduling model used in the hybrid cloud deployment cluster Spark job scheduling method provided in the first embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the structure of a hybrid cloud deployment cluster Spark job scheduling device provided in Embodiment 2 of the present invention;

[0039] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0041] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0042] Embodiment 1

[0043] Figure 1 Flow chart of the hybrid cloud deployment cluster Spark job scheduling method provided in the first embodiment of the present invention. This embodiment can be applied to the situation of efficiently solving the problem of concurrent scheduling of Spark jobs. The method can be executed by the hybrid cloud deployment cluster Spark job scheduling device provided in the embodiment of the present invention, which can be implemented by hardware and / or software, and can generally be integrated into a computer device. Figure 1 As shown, the specific steps include:

[0044] S11. Obtain current job information, virtual machine information, and stage information in real time.

[0045] S12. Based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as an optimization goal to generate current scheduling information.

[0046] S13. Schedule the executor to execute according to the scheduling information.

[0047] The hybrid cloud deployment cluster Spark job scheduling method proposed in this embodiment can be called CA-PSO method, which is a meta-heuristic scheduling method based on particle swarm optimization algorithm (PSO), and its goal is to reduce the virtual machine usage cost of the entire cluster as much as possible in the hybrid cloud deployment cluster. For this goal, the dynamic job scheduling problem between local VM and cloud VM can be described first. In a hybrid cloud deployment cluster, the resource capacity of local VM and cloud VM instances can be equal, but by configuring different types of VM instances, the cost can be reduced more effectively, because jobs with lower resource requirements can be assigned to small VMs to achieve cost optimization. In the hybrid cloud deployment Spark cluster environment, the complexity of job scheduling is reflected in the dependencies between each Stage and its dynamically changing resource requirements. Each Spark job is organized in the form of DAG, where the node represents the stage of the job, and the data flow between the stages constitutes the execution process of the job. Considering the differences in resource requirements between different stages, such as the different requirements for CPU resources of Stage-1 and Stage-2 in Job-1, this places high demands on the resource scheduling capabilities of the cluster. For example, Stage-1 of Job-1 requires 2 CPUs, while Stage-2 of Job-1 requires 4 CPUs. An effective scheduling strategy needs to be able to evaluate the resource requirements of each Stage in real time and intelligently allocate execution resources among VM nodes in the hybrid cloud deployment cluster environment to ensure the smooth execution of the job and optimize the overall efficiency of the cluster.

[0048] A Spark cluster can run one or more parallel applications, which contain one or more jobs, which together form a DAG of processing stages. In the current Spark job scheduling, the main problem is to find one or more available VMs for each Stage in the DAG of each Spark job. In addition, the resource requirements of all Stages placed in the corresponding VM must be constrained by the resource capacity of the VM. At the same time, consider the situation where multiple users submit jobs, that is, multiple jobs from different users can run on the cluster at the same time. Therefore, if one or more Stages from different jobs are placed in the same VM, the resource capacity constraint of the VM must be satisfied by considering all Stages of different sizes from multiple jobs. To this end, the resource availability of the VM can be tracked dynamically instead of just considering the resource capacity. The resource capacity and resource availability of the VM are initially the same. As jobs are continuously scheduled, one or more Stages of one or more jobs are placed in the VM. The resource availability of the VM will decrease over time, while the resource capacity of the VM remains unchanged. In addition, as the Stages in the VM complete execution, the resource availability of the VM will gradually increase. This dynamic change reflects the actual situation of job execution and can more accurately meet the resource matching requirements between jobs and VMs.

[0049] For the current Spark job scheduling, the current job information, virtual machine (VM) information and stage (Stages) information can be obtained in real time. Based on this information, the particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as the optimization goal, and the current scheduling information is optimized, so as to schedule the executor to execute according to the scheduling information. Among them, the particle swarm optimization algorithm, as an evolutionary computing technology inspired by the predation behavior of bird flocks, shows strong robustness and search capabilities. The algorithm is based on the observation of animal cluster behavior. Through information sharing and collaboration between individuals, it guides the entire group to evolve from disorder to order in the problem space, and finally finds the global optimal solution. It is not only good at handling multi-dimensional, nonlinear, and non-convex optimization problems, but also has the ability to merge with other algorithms, thereby further improving search efficiency and performance.

[0050] Optionally, the scheduling information includes: the stage scheduling order provided to the directed acyclic graph scheduler, and the corresponding relationship between tasks and executors provided to the task scheduler. Specifically, the scheduling model used by the CA-PSO method is as follows: Figure 2As shown. The traditional Spark master manages application execution and monitors the health of many workers. Each worker allocates resources among multiple executors. Executors are created for specific applications and remain associated with the application. The application handles its own work scheduling for the executors. Once the application is completed, its executor will terminate. In order to apply the CA-PSO method to Spark, a cost-oriented CA-PSO scheduler can be set accordingly. The CA-PSO scheduler can dynamically obtain job information and VM information from the Spark master, and can obtain stage information from the directed acyclic graph scheduler (DAGScheduler). After calculating the scheduling information, the CA-PSO scheduler can respond to the DAG Scheduler with the stage scheduling order, and can provide the task-executor correspondence to the Task Scheduler. At the same time, the Task Scheduler can also obtain task information from the DAG Scheduler, so as to perform scheduling based on task information and the correspondence between tasks and executors.

[0051] On the basis of the above technical solution, optionally, based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as the optimization goal to generate the current scheduling information, including: creating a particle swarm, and randomly initializing the position and speed of each particle, each particle representing a possible placement scheme in the hybrid cloud deployment cluster for the job stage; dividing the particle swarm into multiple groups, and determining the optimal placement scheme within the group with the lowest virtual machine usage cost in each group; updating the position and speed of each particle in the corresponding group according to the optimal placement scheme within the group, and generating a new placement scheme for the corresponding particle based on the updated position and speed; iteratively re-determining the optimal placement scheme within each group based on the new placement scheme until a preset termination condition is reached, determining the global optimal placement scheme according to the final optimal placement scheme within the group, and generating the scheduling information based on the global optimal placement scheme.

[0052] Specifically, the CA-PSO method can first ensure efficient processing of continuously submitted Spark job requests through job queue management, resource pre-allocation evaluation, and preliminary job priority setting, so as to ensure that multiple jobs can run in parallel and receive scheduling, thereby reducing job waiting time and improving the overall throughput of the cluster. On this basis, for the current job, a particle swarm can be created first, in which each particle represents a possible placement scheme for the job stage in the hybrid cloud deployment cluster. These placement schemes are randomly generated to ensure that the algorithm can cover a wide range of solution spaces and increase the possibility of finding the global optimal solution. For each particle, the number of stages currently exists, that is, the number of position dimensions, and the value of each dimension represents which VM a stage should be placed in. At the same time, the position and speed of each particle are also initialized to prepare for the subsequent iterative optimization process. Specifically, the speed and position of each particle can be randomly assigned, and the maximum value of the speed and position is the number of VMs. The randomness of the initialization process is the key, which helps the algorithm escape from the local optimal solution to explore more potential high-quality solutions.

[0053] Then, the particle swarm is iterated and updated, and the particle swarm is divided into M groups. Each group performs iterative optimization independently. The grouping strategy is helpful for parallel computing and accelerating convergence. In each group, particles compare their respective placement plans to select the best placement plan in the group. Different placement plans will have a significant impact on the scheduling results. In order to guide particles to better explore solutions in the direction of the optimization goal, a greedy method can be used, that is, the placement plan of the particle with the lower virtual machine usage cost is considered to be better.

[0054] For each group, after obtaining the optimal placement plan within the group, the position and speed of each particle in the corresponding group can be updated according to the optimal placement plan within the group to move it to a better solution space. Specifically, the global optimal placement plan can be determined first according to the optimal placement plan within each group, and then for a certain group, the speed of the corresponding particle can be updated according to the global optimal placement plan and the position of the optimal placement plan within its group and the current speed of each particle in the group, and then the position of the corresponding particle can be updated according to the old position and new speed of each particle. After the update is completed, a new placement plan for the corresponding particle can be generated based on the updated position and speed to prepare for the next round of iteration. The above process can then be repeated until a preset termination condition is reached, wherein the preset termination condition can be that a preset maximum number of iterations is reached or there is no obvious improvement in the solution for multiple consecutive iterations, etc. After the preset termination condition is reached, the global optimal placement plan can be determined according to the final optimal placement plan within the group, and scheduling information is generated based on the global optimal placement plan.

[0055] Further optionally, after the position and speed of each particle in the corresponding group is updated according to the optimal placement plan in the group, before the new placement plan of the corresponding particle is generated based on the updated position and speed, it also includes: using a multi-point crossover strategy to cross-update the particles in each group. Specifically, before updating the placement plan, a multi-point crossover strategy can be used to cross-update the particles in the group to increase the diversity of the population, which helps to combine the advantages of different particles and generate new and potentially better solutions. Specifically, the positions of the particles in the group can be crossed, that is, the positions of two particles are exchanged each time they cross to obtain a new position. Optionally, the probability of crossover update within the group is 15%.

[0056] On the basis of the above technical solution, optionally, the optimization target is specifically:

[0057] min:Cost total =Cost L +Cost C ;

[0058]

[0059] Among them, Cost total Indicates the virtual machine usage cost of the entire cluster. L Indicates the cost of using the local virtual machine. C represents the cost of using cloud virtual machines, M represents the total number of local virtual machines, and x j Indicates the usage status of the jth local virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the jth local virtual machine, T C represents the estimated completion time of the current job, N represents the total number of cloud virtual machines, and y k Indicates the usage status of the kth cloud virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the k-th cloud virtual machine, Indicates the estimated completion time of the current job when cloud virtual machines are used.

[0060] Specifically, for the optimization goal of the CA-PSO method, a local resource model and a cloud resource model can be constructed. For the local resource model, let the local virtual machine set L = {1, 2, ..., M}, where M is the total number of local virtual machines, and 1 ≤ j ≤ M is the jth virtual machine deployed locally. For the local virtual machine usage cost of the current job, Cost L , defines a binary decision variable x j To describe the usage status of the local virtual machine, it can be:

[0061]

[0062] Another binary decision variable u is defined, ij which is used to indicate whether the i-th stage is placed in the j-th local virtual machine. Specifically, it can be:

[0063]

[0064] Cost L The T in C is the estimated completion time of the current job, and the T of each job can be calculated through the job information. C 。 Each stage of each job can be placed only on the local VM or in a mixed manner on the local VM and the cloud VM. However, if the stage is selected to be placed on any cloud VM, due to data transfer and network latency, the T C placed on the cloud VM will be higher than the T C placed only on the local VM. C Considering the two cases separately, T

[0065]

[0066] where represents the estimated completion time when placed only on the local VM, represents the estimated completion time in the mixed setting on the local VM and the cloud VM. That is, if the sum of all local placements is equal to E, it indicates that the current job is running entirely on the local VM.

[0067] For the cloud resource model, let the set of cloud virtual machines C = {1, 2,..., N}, where N is the total number of cloud virtual machines, and 1 ≤ k ≤ N is the k-th virtual machine for cloud deployment. For the cost Cost C of using cloud virtual machines for the current job, a binary decision variable y k is defined to indicate the usage status of the cloud virtual machine. Specifically, it can be:

[0068]

[0069] Another binary decision variable v is defined, ik which is used to indicate whether the i-th stage is placed in the k-th cloud virtual machine. Specifically, it can be:

[0070]

[0071] where, considering that the total number of local VMs may be limited, cloud VMs will be used for calculation, and it can be assumed that M < N, and P L < P C, that is, the cost of deploying the same type of VM instance locally is lower than deploying it in the cloud, because the cost of using local VMs is usually lower than that in the cloud.

[0072] Based on the above local resource model and cloud resource model, the optimization problem can be expressed as: when scheduling the current job, while ensuring that the job deadline is met, the usage cost of the entire cluster is minimized. The total cost consists of the costs of all VMs locally and in the cloud.

[0073] Further optionally, the optimization objective is based on the following constraints:

[0074]

[0075] Where E represents the total number of stages required for the current job, u ij Indicates whether the i-th stage is placed in the j-th local virtual machine, 1 if yes, 0 if no. represents the CPU resources required for the i-th stage, represents the currently available CPU resources in the jth local virtual machine, v ik Indicates whether the i-th stage is placed in the k-th cloud virtual machine, 1 if yes, 0 if no. Indicates the CPU resources currently available in the k-th cloud virtual machine.

[0076] Specifically, as mentioned above, the total resource demand of all stages in a VM should not exceed the total resource capacity of the VM, then there are local resource capacity constraints and cloud resource capacity constraints respectively. If the current resource capacity constraints are met, the job scheduler can select one or more stages in the current job to be placed in the same VM. In addition, the stage placement constraint must also be met, that is, each stage of each job can only be placed in one VM. All constraints can be expressed as the above constraint condition formula. Correspondingly, the optimization problem can be expressed as minimizing the virtual machine usage cost of the entire cluster under the premise of meeting various constraints. By optimizing based on the above optimization objectives under the above constraints, it is possible to fully utilize the diverse pricing strategies of VM instances in a hybrid cloud environment, optimize the use of VM resources, and further reduce the operating costs of the cluster.

[0077] The CA-PSO method proposed in this embodiment can show the minimum VM usage cost of the hybrid cluster, because under the continuous search of the particle swarm, the method continuously selects the optimal placement scheme that can reduce the duration of the job. Compared with GIO, CA-PSO can reduce the VM usage cost of the hybrid cluster by about 10%, and compared with Decima, it can reduce it by 15%. In addition, CA-PSO can adapt to different pricing models and different Spark job scenarios, and has strong scalability.

[0078] The technical solution provided by the embodiment of the present invention first obtains the current job information, virtual machine information and stage information in real time, and then based on this information, uses the particle swarm optimization algorithm to minimize the virtual machine usage cost of the entire cluster as the optimization goal, generates the current scheduling information, and then schedules the executor to execute according to the obtained scheduling information. By combining the job information, virtual machine information and stage information, and based on the meta-heuristic particle swarm optimization algorithm, and then through the iterative search of the particle swarm, it is possible to accurately customize the optimal scheduling solution for the DAG of each Spark job in a multi-job environment, thereby effectively reducing the overall operating cost of the hybrid cloud deployment cluster.

[0079] Embodiment 2

[0080] Figure 3 This is a structural diagram of a hybrid cloud deployment cluster Spark job scheduling device provided in Embodiment 2 of the present invention. The device can be implemented by hardware and / or software, and can generally be integrated into a computer device to execute the hybrid cloud deployment cluster Spark job scheduling method provided in any embodiment of the present invention.

[0081] like Figure 3 As shown, the device comprises:

[0082] The information acquisition module 31 is used to obtain the current job information, virtual machine information and stage information in real time;

[0083] A scheduling information generating module 32 is used to generate current scheduling information based on the job information, the virtual machine information and the stage information by using a particle swarm optimization algorithm with the optimization goal of minimizing the virtual machine usage cost of the entire cluster;

[0084] The task execution module 33 is used to schedule the executor to execute according to the scheduling information.

[0085] The technical solution provided by the embodiment of the present invention first obtains the current job information, virtual machine information and stage information in real time, and then based on this information, uses the particle swarm optimization algorithm to minimize the virtual machine usage cost of the entire cluster as the optimization goal, generates the current scheduling information, and then schedules the executor to execute according to the obtained scheduling information. By combining the job information, virtual machine information and stage information, and based on the meta-heuristic particle swarm optimization algorithm, and then through the iterative search of the particle swarm, it is possible to accurately customize the optimal scheduling solution for the DAG of each Spark job in a multi-job environment, thereby effectively reducing the overall operating cost of the hybrid cloud deployment cluster.

[0086] On the basis of the above technical solution, optionally, the scheduling information generating module 32 includes:

[0087] A particle swarm creation unit is used to create a particle swarm and randomly initialize the position and velocity of each particle. Each particle represents a possible placement scheme in the hybrid cloud deployment cluster during the job phase.

[0088] An intra-group optimal determination unit, used for dividing the particle swarm into a plurality of groups, and determining an intra-group optimal placement scheme with the lowest virtual machine usage cost in each group;

[0089] A placement scheme updating unit, used for updating the position and speed of each particle in the corresponding group according to the optimal placement scheme in the group, and generating a new placement scheme for the corresponding particle based on the updated position and speed;

[0090] An iterative optimization unit is used to iteratively redetermine the optimal placement plan within each group based on the new placement plan until a preset termination condition is reached, determine the global optimal placement plan based on the final optimal placement plan within the group, and generate the scheduling information based on the global optimal placement plan.

[0091] On the basis of the above technical solution, optionally, the scheduling information generating module 32 further includes:

[0092] A crossover updating unit is used to cross-update the particles in each group using a multi-point crossover strategy after the position and speed of each particle in the corresponding group is updated according to the optimal placement plan in the group and before a new placement plan for the corresponding particle is generated based on the updated position and speed.

[0093] On the basis of the above technical solution, optionally, the optimization target is specifically:

[0094] min:Cost total =Cost L +Cost C ;

[0095]

[0096] Among them, Cost total Indicates the virtual machine usage cost of the entire cluster. L Indicates the cost of using the local virtual machine. C represents the cost of using cloud virtual machines, M represents the total number of local virtual machines, and x j Indicates the usage status of the jth local virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the jth local virtual machine, T C represents the estimated completion time of the current job, N represents the total number of cloud virtual machines, and y k Indicates the usage status of the kth cloud virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the k-th cloud virtual machine, Indicates the estimated completion time of the current job when cloud virtual machines are used.

[0097] On the basis of the above technical solution, optionally, the optimization target is based on the following constraints:

[0098]

[0099] Where E represents the total number of stages required for the current job, u ij Indicates whether the i-th stage is placed in the j-th local virtual machine, 1 if yes, 0 if no. represents the CPU resources required for the i-th stage, represents the currently available CPU resources in the jth local virtual machine, v ik Indicates whether the i-th stage is placed in the k-th cloud virtual machine, 1 if yes, 0 if no. Indicates the CPU resources currently available in the k-th cloud virtual machine.

[0100] On the basis of the above technical solution, optionally, the scheduling information includes: a stage scheduling sequence provided to a directed acyclic graph scheduler, and a correspondence between tasks and executors provided to a task scheduler.

[0101] The hybrid cloud deployment cluster Spark job scheduling device provided in the embodiment of the present invention can execute the hybrid cloud deployment cluster Spark job scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0102] It is worth noting that in the above-mentioned embodiment of the hybrid cloud deployment cluster Spark job scheduling device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0103] Embodiment 3

[0104] Figure 4 The schematic diagram of the structure of the computer device provided for the third embodiment of the present invention shows a block diagram of an exemplary computer device suitable for implementing the implementation mode of the present invention. Figure 4 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 4 As shown, the computer device includes a processor 41, a memory 42, an input device 43 and an output device 44; the number of processors 41 in the computer device can be one or more. Figure 4Taking a processor 41 as an example, the processor 41, memory 42, input device 43 and output device 44 in the computer device can be connected through a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0105] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the hybrid cloud deployment cluster Spark job scheduling method in the embodiment of the present invention (for example, the information acquisition module 31, the scheduling information generation module 32 and the task execution module 33 in the hybrid cloud deployment cluster Spark job scheduling device). The processor 41 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 42, that is, realizing the above-mentioned hybrid cloud deployment cluster Spark job scheduling method.

[0106] The memory 42 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The input device 43 can be used to obtain currently required information in real time, and generate key signal input related to user settings and function control of the computer device, etc. The output device 44 can be used to schedule execution based on scheduling information, etc.

[0108] Embodiment 4

[0109] Embodiment 4 of the present invention further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, is used to execute a hybrid cloud deployment cluster Spark job scheduling method, the method comprising:

[0110] Get current job information, virtual machine information, and stage information in real time;

[0111] Based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as an optimization goal to generate current scheduling information;

[0112] The executor is scheduled to execute according to the scheduling information.

[0113] The storage medium may be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system, which is connected to the computer system via a network (such as the Internet). The second computer system may provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that may be executed by one or more processors.

[0114] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the method operations described above, and the computer executable instructions can also execute related operations in the hybrid cloud deployment cluster Spark job scheduling method provided in any embodiment of the present invention.

[0115] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0117] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0118] Embodiment 5

[0119] Embodiment 5 of the present invention further provides a computer program product, which includes a computer program (also referred to as code, instruction), which can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the hybrid cloud deployment cluster Spark job scheduling method provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.

[0120] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A hybrid cloud deployment cluster Spark job scheduling method, characterized in that: include: Get current job information, virtual machine information, and stage information in real time; Based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as an optimization goal to generate current scheduling information; The executor is scheduled to execute according to the scheduling information.

2. The hybrid cloud deployment cluster Spark job scheduling method according to claim 1 is characterized in that: Based on the job information, the virtual machine information and the stage information, a particle swarm optimization algorithm is used to minimize the virtual machine usage cost of the entire cluster as an optimization goal to generate current scheduling information, including: Create a particle swarm and randomly initialize the position and velocity of each particle. Each particle represents a possible placement scheme in the hybrid cloud deployment cluster during the job phase. Dividing the particle swarm into a plurality of groups, and determining an optimal placement scheme in each group with the lowest virtual machine usage cost; updating the position and velocity of each particle in the corresponding group according to the optimal placement scheme in the group, and generating a new placement scheme for the corresponding particle based on the updated position and velocity; Iteratively re-determine the optimal placement plan within each group based on the new placement plan until a preset termination condition is reached, determine the global optimal placement plan based on the final optimal placement plan within the group, and generate the scheduling information based on the global optimal placement plan.

3. The hybrid cloud deployment cluster Spark job scheduling method according to claim 2 is characterized in that: After updating the position and speed of each particle in the corresponding group according to the optimal placement plan in the group, and before generating a new placement plan for the corresponding particle based on the updated position and speed, the method further includes: A multi-point crossover strategy is used to cross-update the particles in each group.

4. The hybrid cloud deployment cluster Spark job scheduling method according to claim 1, characterized in that: The optimization objectives are specifically: min:Cost total =Cost L +Cost C ; Among them, Cost total Indicates the virtual machine usage cost of the entire cluster. L Indicates the cost of using the local virtual machine. C represents the cost of using cloud virtual machines, M represents the total number of local virtual machines, and x j Indicates the usage status of the jth local virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the jth local virtual machine, T C represents the estimated completion time of the current job, N represents the total number of cloud virtual machines, and y k Indicates the usage status of the kth cloud virtual machine. If it is in use, it is 1; if it is not in use, it is 0. represents the unit price of the k-th cloud virtual machine, Indicates the estimated completion time of the current job when cloud virtual machines are used.

5. The hybrid cloud deployment cluster Spark job scheduling method according to claim 4 is characterized in that: The optimization objective is based on the following constraints: Where E represents the total number of stages required for the current job, u ij Indicates whether the i-th stage is placed in the j-th local virtual machine, 1 if yes, 0 if no. represents the CPU resources required for the i-th stage, represents the currently available CPU resources in the jth local virtual machine, v ik Indicates whether the i-th stage is placed in the k-th cloud virtual machine, 1 if yes, 0 if no. Indicates the CPU resources currently available in the k-th cloud virtual machine.

6. The hybrid cloud deployment cluster Spark job scheduling method according to claim 1, characterized in that: The scheduling information includes: a stage scheduling sequence provided to a directed acyclic graph scheduler, and a corresponding relationship between tasks and executors provided to a task scheduler.

7. A hybrid cloud deployment cluster Spark job scheduling device, characterized in that: include: The information acquisition module is used to obtain the current job information, virtual machine information and stage information in real time; A scheduling information generation module, configured to generate current scheduling information based on the job information, the virtual machine information and the stage information by using a particle swarm optimization algorithm with the optimization goal of minimizing the virtual machine usage cost of the entire cluster; The task execution module is used to schedule the executor to execute according to the scheduling information.

8. A computer device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the hybrid cloud deployment cluster Spark job scheduling method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the hybrid cloud deployment cluster Spark job scheduling method as described in any one of claims 1-6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for scheduling Spark jobs in a hybrid cloud deployment cluster is implemented.

Images