Server-free MapReduce multi-target scheduling optimization method

Abstract

The invention relates to the technical field of cloud computing and distributed computing scheduling, in particular to a server-free MapReduce multi-target scheduling optimization method. Comprising the following steps: constructing a directed acyclic graph, coding a resource allocation decision into a solution from a source node to a target node, generating an initial population by adopting a hybrid initialization strategy, evaluating fitness, calculating a congestion distance of a solution in each non-dominated leading edge, and sequentially executing tournament selection, a self-adaptive crossover strategy and a topological repair variation strategy. And periodically executing a solution injection strategy guided by heuristic search, carrying out crowding distance selection, updating the global Pareto optimal solution set, outputting the current global Pareto optimal solution set when the time is up, and otherwise, returning to iteration. The method has the positive effects that the execution cost and the execution time of the MapReduce operation are balanced, the global Pareto optimal solution set quality of the multi-target scheduling scheme is improved, and the convergence speed and the robustness of the algorithm are enhanced.

Description

TECHNICAL FIELD

[0001] The application relates to the field of cloud computing and distributed computing scheduling, and particularly relates to an optimization method for multi-target resource scheduling of MapReduce data-intensive workflow under a serverless computing architecture. BACKGROUND

[0002] Serverless computing is an important paradigm of cloud computing, which provides users with the core advantages of automatic scaling, actual use-based billing and maintenance-free in the form of function as a service. With the continuous growth of big data processing demand, serverless platforms begin to deploy MapReduce distributed computing framework. In serverless MapReduce jobs, the task processing unit Mapper in the Map phase and the task processing unit Reducer in the Reduce phase are implemented in the form of serverless functions, and the execution process of the serverless MapReduce job includes multiple Mapper functions, a coordinator function and multiple Reducer functions. In the traditional cloud computing environment, the research on MapReduce scheduling mainly focuses on single target optimization, such as minimizing execution time or reducing execution cost. In actual application, the scheduling system needs to consider multiple conflicting targets at the same time, including resource efficiency, execution time, energy consumption and operating cost. Multi-objective optimization algorithms provide a direction for solving multi-objective conflict problems, and representative algorithms include non-dominated sorting genetic algorithm, decomposition-based multi-objective evolutionary algorithm, multi-objective particle swarm optimization algorithm and multi-objective grey wolf optimization algorithm.

[0003] However, serverless MapReduce job scheduling faces the following technical defects: first, there is a multi-objective conflict between the execution cost and the execution time of the MapReduce job. On the premise of meeting the resource constraints, pursuing a lower execution cost requires reducing the number of functions or selecting a smaller memory configuration, which will lead to an increase in execution time, and vice versa, pursuing a lower execution time requires more concurrent functions or higher resource configuration, which will increase the execution cost. Second, the traditional multi-objective evolutionary algorithm has difficulty in converging to the optimal Pareto front within a limited execution time when dealing with serverless MapReduce job scheduling problems, and it is easy to produce solutions that do not meet the resource constraint conditions in the evolutionary search process. Third, although heuristic search can quickly find the optimal solution, it is only suitable for single-target optimization scenarios and cannot directly handle multi-objective optimization problems. Therefore, how to design a multi-objective optimization framework to accelerate convergence and improve the quality of the global Pareto optimal solution set while ensuring that the solution meets the resource constraint conditions is a problem that needs to be solved in the optimization of MapReduce job scheduling in a serverless environment. SUMMARY

[0004] The serverless MapReduce multi-objective scheduling optimization method provided by the application solves the technical problems of multi-objective conflict between execution cost and execution time in the MapReduce job scheduling process under the serverless environment, slow convergence speed of traditional multi-objective evolutionary algorithm and solution not meeting resource constraint conditions, so as to balance the execution cost and execution time of the MapReduce job, improve the convergence speed of the algorithm and the uniformity of the global Pareto optimal solution set distribution.

[0005] The serverless MapReduce multi-objective scheduling optimization method provided by the application is characterized in that the method comprises the following steps:

[0006] Step 1, initialize the operation parameters of the heuristic search guided non-dominated sorting genetic algorithm, including population size, crossover rate, mutation rate and heuristic search guided injection interval;

[0007] Step 2, construct a directed acyclic graph of resource allocation of the serverless MapReduce job, and encode the resource allocation decision as a solution from a source node to a target node;

[0008] Step 3, generate an initial population by using a hybrid initialization strategy;

[0009] Step 4, evaluate the fitness of each solution in the current population by using an objective function of minimizing task completion time and an objective function of minimizing execution cost, and take the current population as a parent population;

[0010] Step 5, perform non-dominated sorting on the current population, initialize the dominance count based on the dominance relationship, iteratively construct the non-dominated front and calculate the crowding distance of the solutions in each non-dominated front;

[0011] Step 6, sequentially perform tournament selection, adaptive crossover strategy and topology repair mutation strategy and generate a child population;

[0012] Step 7, periodically perform a heuristic search guided solution injection strategy, inject a single-objective optimal solution into the child population and update the child population;

[0013] Step 8, merge the parent population and the child population, perform crowding distance selection on the merged temporary population and generate a new population;

[0014] Step 9, update the global Pareto optimal solution set;

[0015] Step 10, judge whether the time limit is reached, if yes, output the current global Pareto optimal solution set and the total execution cost and total execution time corresponding to the current global Pareto optimal solution set, otherwise return to step 4 for iteration.

[0016] Further, in step 2, the construction method of the directed acyclic graph model is to encode the resource allocation decision into the structure of the solution, model each solution as a path from the source node to the target node in the directed acyclic graph, denotes the source node, denotes the target node, and other nodes in the directed acyclic graph are denoted by , wherein, denotes the stage index of the resource decision stage, denotes the node index within the resource decision stage, denotes the th node within the th stage, , the value of is 1, 2, 3, 4, or 5, 1 corresponds to the Mapper memory configuration stage; 2 corresponds to the Mapper number stage; 3 corresponds to the number of input objects of the Reducer stage; 4 corresponds to the Coordinator memory configuration stage; 5 corresponds to the Reducer memory configuration stage; the solution encoding sequence , respectively denote the nodes corresponding to each resource decision stage of the path, and each path corresponds to only one node in each resource decision stage; each edge in the directed acyclic graph connects two nodes of adjacent stages, denotes the transition from the resource configuration selection of one resource decision stage to the resource configuration selection of another resource decision stage, and each edge stores the execution time and execution cost of the corresponding resource configuration; the validity of the solution is judged based on the connection of all adjacent nodes in the directed acyclic graph through edges, and the cumulative execution time and cumulative execution cost do not exceed the preset threshold, wherein the cumulative execution cost represents the sum of the execution costs of all edges on the path from the source node to the current node in the path search process, and the cumulative execution time represents the sum of the execution times of all edges on the path from the source node to the current node in the path search process, when the path search is completed, i.e., the path reaches the target node from the source node , the cumulative execution cost is equal to the total execution cost, and the cumulative execution time is equal to the total execution time.

[0017] Furthermore, in step 3, the hybrid initialization strategy includes a heuristic search-guided process and a constraint-aware random generation process. In the heuristic search-guided process, heuristic search is used to generate solutions with minimum execution cost and minimum execution time. In the constraint-aware random generation process, different solutions are constructed through depth-first search and dynamic pruning methods.

[0018] When using heuristic search to generate the solution with minimum execution cost, an adaptive threshold approach is employed. The process involves starting from the source node... Initially, maintain an open list and a closed list. The open list stores nodes to be expanded, and the closed list stores expanded nodes. Each time, select the node with the smallest evaluation function value from the open list for expansion. The evaluation function is defined as follows:

[0019]

[0020] in Represents a node The evaluation function value, This represents a node in a directed acyclic graph. Indicates from the source node To the node The actual implementation cost Indicates from node To the target node The estimated execution cost, The relaxation factor is dynamically adjusted when the expanded node is the target node. When the backtracking path yields the solution with the minimum execution cost;

[0021] When using heuristic search to generate a solution with the minimum execution time, a reverse search approach is employed. The process involves starting from the target node... To the source node Perform a reverse search, maintaining an open list and a closed list. Each time, select the node with the shortest cumulative execution time from the open list for expansion. When the expanded node is the source node... When the time is right, the backtracking path yields the solution with the minimum execution time;

[0022] The specific process of depth-first search is as follows: starting from the source node... Begin by initializing the current node as the source node. The initial path is empty, and the source node is set. Add a path when the current node is not equal to the target node. When the current node is empty, obtain the adjacency list of the current node. The adjacency list stores all edges originating from the current node. If the adjacency list is empty, terminate the generation of the current path. Randomly and evenly select an edge from the adjacency list, update the target node connected to the selected edge to the current node, and add the execution cost and execution time of the current node and edge to the path.

[0023] The specific process of the dynamic pruning method is as follows: after each node expansion, calculate the number of nodes from the source node. The cumulative execution cost and cumulative execution time up to the current node, when the cumulative execution cost exceeds the budget limit. Or the cumulative execution time exceeds the deadline. If the current path cannot be expanded, backtrack to the last node in the path that satisfies the resource constraints. Remove the tried edges from the adjacency list of the last node that satisfies the resource constraints, and randomly select an edge that satisfies the resource constraints again for expansion. If no edge that satisfies the resource constraints can be found after backtracking, abandon the current path and start again from the source node. Start generating a new path.

[0024] Furthermore, resource constraints include function concurrency limits, temporary storage limits, quality of service constraints, and selection variable constraints, among which,

[0025] The mathematical expression for the function concurrency limit is:

[0026]

[0027] In the formula, This indicates the number of serverless functions used in the Map phase. This indicates the number of steps in the Reduce phase. Indicates the Reduce phase. The number of Reducers in each step This indicates the maximum concurrent resource limit allowed by the serverless platform;

[0028] The mathematical expression for the temporary storage limit is:

[0029]

[0030] In the formula, Indicates the index of the Map task. Indicates the first The size of intermediate results for each Map task, in MB. This indicates platform limitations;

[0031] Service quality constraints include task completion time constraints and execution cost constraints. The mathematical expression for the task completion time constraint is:

[0032]

[0033] In the formula, Indicates the total execution time. Indicates the deadline;

[0034] The mathematical expression for the execution cost constraint is:

[0035]

[0036] In the formula, Indicates the total execution cost. The mathematical expression representing budget constraints and selection variable constraints is:

[0037]

[0038]

[0039] in Indicates the number of optional configurations. Indicates the selection of the first Indicator variables for various types of serverless functions Indicates the selection of the first This memory type is used for the indicator variable of the Reducer. and Use binary variables to ensure the uniqueness of the schedule.

[0040] Furthermore, in step 4, the mathematical expression for the objective function that minimizes the task completion time is:

[0041]

[0042] in Let represent the objective function that minimizes the task completion time. Indicates the total execution time. Indicates the execution time of the Map phase. Indicates the execution time of the coordinator phase. Indicates the execution time of the Reduce phase;

[0043] The mathematical expression for the objective function that minimizes execution cost is:

[0044]

[0045] in This represents the objective function that minimizes the execution cost. Indicates the total execution cost. Indicates the cost of serverless function calls. Indicates the execution cost of a serverless function. Indicates the request cost of the object storage system. The object storage system storage cost represents intermediate data. The serverless function call cost refers to the fixed cost incurred when calling a serverless function. The serverless function execution cost refers to the cost calculated based on the execution time and memory configuration during the execution of a serverless function. The object storage system refers to a distributed storage system used to store intermediate data. The object storage system request cost refers to the request cost incurred when accessing the object storage system. The object storage system storage cost refers to the storage cost of intermediate data in the object storage system.

[0046] Furthermore, in step 5, the non-dominated sorting method involves initializing the dominant solution list and dominance count for each solution, comparing any combination of two solutions in the current population, and if the solution... It is superior to or equal to the solution in both the objective function of minimizing task completion time and the objective function of minimizing execution cost. It outperforms at least one of the objective functions: minimizing task completion time and minimizing execution cost. ,but Dominate ,Will join in The list of dominant solutions will The dominance count is incremented by 1. Solutions with a dominance count of 0 are added to the first non-dominated frontier. A dominance count of 0 indicates that a solution is not dominated by any other solution. The next non-dominated frontier is calculated iteratively. All solutions dominated by each solution in the current non-dominated frontier are traversed. The dominance count of each traversed solution is decremented by 1. When the dominance count of a traversed solution drops to 0, the traversed solution is added to the next non-dominated frontier. The first non-dominated frontier has the lowest level, and the level of the next non-dominated frontier increases.

[0047] The crowding distance is calculated as follows: for each non-dominated frontier, sort the solutions according to the values ​​of the objective function that minimizes task completion time and the objective function that minimizes execution cost, and obtain a sorted sequence. Set the crowding distance of the solution with the smallest or largest value on the objective function that minimizes task completion time in the non-dominated frontier to infinity, and set the crowding distance of the solution with the smallest or largest value on the objective function that minimizes execution cost in the non-dominated frontier to infinity. The crowding distance of other solutions in the non-dominated frontier is the sum of the normalized distances of the solutions at adjacent positions in the sorted sequence on the objective functions that minimize task completion time and minimize execution cost.

[0048] Furthermore, in step 6, the tournament selects a preset number of solutions from the current population for comparison, and selects the solution with a lower level of non-dominant front and a larger crowding distance as the parent solution.

[0049] The adaptive crossover strategy employs a crossover method based on adaptive crossover point selection. At the selected crossover point, it swaps the node configurations of the two parent solutions and recalculates the total execution cost and total execution time of the solution. The formula for calculating the crossover point selection probability is as follows:

[0050]

[0051] In the formula, Indicates the selection of a location As the probability of choosing the intersection point For the intersection location index, For adaptive parameters, The length of the solution;

[0052] Topology repair mutation strategies include the selection of mutated nodes, the determination of nodes, and the repair process, targeting the decoded sequence. In the process of selecting mutation nodes, from a uniform distribution Randomly select a mutated node , This represents the total number of nodes in the decoded sequence; during the node determination process, it starts from the mutated nodes. A new edge is randomly selected from the adjacency list of the predecessor node to replace it. The edge replacement must satisfy the topological continuity constraint, that is, there exists an edge from the predecessor node to the target node of the new edge, and the target node of the new edge is not equal to the mutated node. The target node of the new edge represents the node that the new edge points to; during the repair process, if the new edge causes a mutated node in the decoded sequence... The following part is related to the target node. If connectivity breaks or topological continuity constraints are violated, a repair mechanism is triggered to check for a connection from the disconnected node to the target node. If a path does not exist, identify and remove nodes that do not meet the resource constraints. Starting from the disconnected node, select edges that meet the resource constraints from the disconnected node's adjacency list to construct a path to the next node, and so on, until the target node is reached. .

[0053] Furthermore, in step 7, the specific process of the heuristic search-guided uninjection strategy is as follows: every... The offspring population is non-dominated and sorted. From the first non-dominated front, solutions are selected in descending order of crowding distance, and the solution with the largest crowding distance is chosen as the representative solution. The injection interval is set according to a preset heuristic search; the total execution cost or total execution time of representative solutions is extracted as a constraint; with the goal of minimizing execution cost, heuristic search is used to obtain the solution with the minimum execution time under the constraint of the total execution cost of representative solutions; with the goal of minimizing execution time, heuristic search is used to search for the solution with the minimum execution cost under the constraint of the total execution time of representative solutions; the generated single-objective optimal solution is injected into the offspring population to replace solutions with higher ranks or smaller crowding distances in the non-dominated sorting, or to supplement regions with smaller crowding distances in the offspring population, thus updating the offspring population.

[0054] Furthermore, in step 8, the crowding distance selection method is as follows: the temporary population is sorted by non-dominated order, the temporary population is divided into multiple non-dominated fronts, the crowding distance of the solution is calculated for each non-dominated front, the solutions are selected in order of increasing level of non-dominated fronts, the crowding distance of each non-dominated front is sorted in order of decreasing level, and the solutions with larger crowding distances are selected first, until the number of selected solutions reaches the preset population size, and a new population is generated.

[0055] Furthermore, in step 9, when updating the global Pareto optimal solution set, the non-dominated front after the current iteration's non-dominated sorting is merged with the historical optimal solution set and non-dominated sorting is performed. The solutions in the first non-dominated front after non-dominated sorting are extracted as the current global Pareto optimal solution set. At the same time, the solution with the minimum execution cost and the solution with the minimum execution time in the current global Pareto optimal solution set are recorded.

[0056] This invention provides a serverless MapReduce multi-objective scheduling optimization method. Addressing the issues raised in the background art regarding the multi-objective conflict between execution cost and execution time of MapReduce jobs in serverless environments, the slow convergence of traditional multi-objective evolutionary algorithms under complex constraints, the tendency to generate solutions that do not meet resource constraints, and the uneven distribution of the global Pareto optimal solution set, this invention constructs a non-dominated sorting genetic algorithm framework that integrates heuristic search. Through a solution injection strategy guided by heuristic search, single-objective optimal solutions are injected into the current population, enhancing the exploration capability of non-dominated region boundaries, improving the uniformity of the global Pareto optimal solution set, and overcoming the drawback of traditional non-dominated sorting genetic algorithms easily getting trapped in local optima in complex graph-constrained problems. In summary, this invention has the positive effects of balancing the execution cost and execution time of MapReduce jobs, improving the quality of the global Pareto optimal solution set of multi-objective scheduling schemes, and enhancing the convergence speed and robustness of the algorithm. Attached Figure Description

[0057] Figure 1 This is a schematic diagram illustrating the implementation process of the present invention;

[0058] Figure 2 This is a schematic diagram of the directed acyclic graph encoding structure of the present invention;

[0059] Figure 3 This is a schematic diagram of the topology repair mutation strategy in this invention;

[0060] Figure 4 This is a comparison chart of the supervolume of the present invention and four existing comparison algorithms under different task scales;

[0061] Figure 5 This is a comparison chart of the reverse generation distance between the present invention and four existing comparison algorithms under different task scales;

[0062] Figure 6 This is a comparison chart of the hypervolume of the present invention and four existing comparison algorithms under different task sizes;

[0063] Figure 7 This is a comparison chart of the reverse generation distance between the present invention and four existing comparison algorithms under different task sizes. Detailed Implementation

[0064] like Figure 1 As shown, the specific implementation process of the serverless MapReduce multi-objective scheduling optimization method provided by the present invention is as follows.

[0065] Step 1: Initialize the heuristically search-guided non-dominated sorting genetic algorithm's running parameters, including population size, crossover rate, mutation rate, and heuristically search-guided injection interval. Calibration experiments are conducted on key parameters to adapt to serverless MapReduce multi-objective scheduling scenarios. The parameter calibration experiments use two multi-objective optimization standard indicators for evaluation: hypervolume and reverse generation distance. A higher hypervolume indicates better solution set quality, while a lower reverse generation distance indicates better solution set convergence. Since hypervolume and reverse generation distance are two conflicting objectives, pursuing a higher hypervolume may lead to an increase in the reverse generation distance, and vice versa. Therefore, parameter selection needs to balance these two objectives. Calibration experiments are conducted on the three key parameters, testing hypervolume and reverse generation distance under different values ​​of population size, crossover rate, and mutation rate. The experimental results are shown in Table 1.

[0066] Table 1 Summary of calibration experimental results for three key parameters

[0067]

[0068] Based on the data analysis in Table 1, since overvolume and reverse generation distance are two conflicting objectives, parameter selection requires a trade-off between the two objectives. For the population size parameter, when the population size is 20, the overvolume is highest (0.98479) but the reverse generation distance is relatively large (0.06813). When the population size is 50, the overvolume is 0.98476 and the reverse generation distance is 0.06787, achieving a better balance between overvolume and reverse generation distance. For the crossover rate parameter, when the crossover rate is 0.9, the overvolume is highest (0.98478) but the reverse generation distance is 0.06787. When the crossover rate is 0.8, the overvolume is 0.98476 and the reverse generation distance is 0.06791, achieving a better balance between overvolume and reverse generation distance. Regarding the mutation rate parameter, when the mutation rate is 0.5, the oversize is the highest (0.98479) but the reverse generation distance is 0.06785. When the mutation rate is 0.1, the oversize is lower (0.98472) and the reverse generation distance is larger (0.06795). When the mutation rate is 0.3, the oversize is 0.98477 and the reverse generation distance is 0.06793. Considering the balance of the two indicators and the robustness of the algorithm, the final parameter configuration is determined as follows: population size of 50, crossover rate of 0.8, and mutation rate of 0.15. For non-critical parameters, the injection interval guided by heuristic search is set to 10 (generations). This parameter controls the execution frequency of the heuristic search-guided de-injection strategy. Setting it to 10 (generations) means that the heuristic search-guided de-injection strategy is executed once every 10 generations. This parameter value is based on the common settings of periodic strategies in multi-objective evolutionary algorithms. Combined with the characteristics of serverless MapReduce multi-objective scheduling scenarios, setting it to 10 (generations) can avoid over-reliance on heuristic search while maintaining the algorithm's convergence speed, ensuring that the algorithm achieves a balance between exploration and utilization.

[0069] Step 2: Construct a directed acyclic graph (DAG) for resource allocation in the serverless MapReduce job. Specifically, the DAG model is constructed by modeling each solution as a source node in the DAG. To the target node The path, Indicates the source node, The target node is represented by [target node], and the other nodes in the directed acyclic graph are [other nodes]. It means that, among them, This represents the stage index of the resource decision-making phase. This represents the node index within the resource decision-making phase. Indicates the first Phase 1 1 node The values ​​are 1, 2, 3, 4, and 5. A value of 1 corresponds to the Mapper memory configuration phase; A value of 2 corresponds to the number of Mappers in the current stage. When the value is 3, it corresponds to the stage of the number of input objects for the Reducer; When the value is 4, it corresponds to the coordinator memory configuration stage; When the value is 5, it corresponds to the Reducer memory configuration stage; decoding sequence. = .

[0070] These represent the nodes corresponding to each resource decision-making stage of the path, where... This indicates the node corresponding to the path during the Mapper memory configuration phase. This indicates the node corresponding to the path in the Mapper count stage. This indicates the node corresponding to the number of input objects in the Reducer's path. This indicates the node corresponding to the path during the coordinator's memory configuration phase. This indicates the node corresponding to the path in the Reducer memory configuration phase. , , , , These represent the sequence numbers of nodes within the corresponding stages, with each path corresponding to only one node in each resource decision stage. For example, in... Figure 2 In the illustrated embodiment, the node set during the Mapper memory configuration phase includes nodes , , , This indicates that the path has four available Mapper memory configuration options during the Mapper memory configuration phase. The value range is 1 to 4; the node set in the Mapper quantity stage contains nodes. , , This indicates that there are 3 available Mapper count options for the path during the Mapper count phase. The value range is 1 to 3; the node set of the input object number stage of the Reducer contains nodes. , , , This indicates that the path has four available options for the number of input objects for the Reducer during the Reducer input object count stage. The value range is 1 to 4; the node set in the coordinator memory configuration phase contains nodes , , , This indicates that there are four available coordinator memory configuration options during the coordinator memory configuration phase. The value range is 1 to 4; the node set in the Reducer memory configuration phase contains nodes. , , , This indicates that there are four available Reducer memory configuration options during the Reducer memory configuration phase. The value range is from 1 to 4. In the directed acyclic graph, each edge connects two nodes in adjacent stages, representing the transition from resource allocation selection in one resource decision stage to resource allocation selection in another. Each edge stores the execution time and execution cost of the corresponding resource allocation. The validity of the solution is judged based on the fact that all adjacent nodes are connected by edges in the directed acyclic graph, and the cumulative execution time and cumulative execution cost do not exceed a preset threshold. Where, the cumulative execution cost represents the time and cost from the source node during the path search process. The cumulative execution time is the sum of the execution costs of all edges on the path to the current node, representing the execution time from the source node during the path search process. The sum of execution times for all edges on the path to the current node. This occurs when the path search is complete, i.e., the path originates from the source node. Reach the target node The cumulative execution cost equals the total execution cost. The cumulative execution time equals the total execution time.

[0071] Step 3: Generate the initial population using a hybrid initialization strategy. This strategy includes a heuristic search-guided process and a constraint-aware random generation process. In the heuristic search-guided process, heuristic search is used to generate solutions with minimum execution cost and minimum execution time. In the constraint-aware random generation process, different solutions are constructed using depth-first search and dynamic pruning methods.

[0072] Specifically, when using heuristic search to generate the solution with the minimum execution cost, an adaptive threshold method is adopted. The process involves starting from the source node... Initially, maintain an open list and a closed list. The open list stores nodes to be expanded, and the closed list stores expanded nodes. Each time, select the node with the smallest evaluation function value from the open list for expansion. The evaluation function is defined as follows:

[0073]

[0074] in Represents a node The evaluation function value, This represents a node in a directed acyclic graph. Indicates from the source node To the node The actual implementation cost Indicates from node To the target node The estimated execution cost, The relaxation factor is dynamically adjusted when the expanded node is the target node. When the backtracking path yields the solution with the minimum execution cost;

[0075] When using heuristic search to generate a solution with the minimum execution time, a reverse search approach is employed. The process involves starting from the target node... To the source node Perform a reverse search, maintaining an open list and a closed list. Each time, select the node with the shortest cumulative execution time from the open list for expansion. When the expanded node is the source node... When the time is right, the backtracking path yields the solution with the minimum execution time;

[0076] The specific process of depth-first search is as follows: starting from the source node... Begin by initializing the current node as the source node. The initial path is empty, and the source node is set. Add a path when the current node is not equal to the target node. When the current node is empty, obtain the adjacency list of the current node. The adjacency list stores all edges originating from the current node. If the adjacency list is empty, terminate the generation of the current path. Randomly and evenly select an edge from the adjacency list, update the target node connected to the selected edge to the current node, and add the execution cost and execution time of the current node and edge to the path.

[0077] The specific process of the dynamic pruning method is as follows: after each node expansion, calculate the number of nodes from the source node. The cumulative execution cost and cumulative execution time up to the current node, when the cumulative execution cost exceeds the budget limit. Or the cumulative execution time exceeds the deadline. If the current path cannot be expanded, backtrack to the last node in the path that satisfies the resource constraints. Remove the tried edges from the adjacency list of the last node that satisfies the resource constraints, and randomly select an edge that satisfies the resource constraints again for expansion. If no edge that satisfies the resource constraints can be found after backtracking, abandon the current path and start again from the source node. Start generating a new path.

[0078] Specifically, resource constraints include function concurrency limits, temporary storage limits, quality of service constraints, and selection variable constraints. The mathematical expression for function concurrency limits is:

[0079]

[0080] In the formula, This indicates the number of serverless functions used in the Map phase. This indicates the number of steps in the Reduce phase. Indicates the Reduce phase. The number of Reducers in each step This indicates the maximum concurrent resource limit allowed by the serverless platform;

[0081] The mathematical expression for the temporary storage limit is:

[0082]

[0083] In the formula, Indicates the index of the Map task. Indicates the first The size of intermediate results for each Map task, in MB. This indicates platform limitations;

[0084] Service quality constraints include task completion time constraints and execution cost constraints. The mathematical expression for the task completion time constraint is:

[0085]

[0086] In the formula, Indicates the total execution time. Indicates the deadline;

[0087] The mathematical expression for the execution cost constraint is:

[0088]

[0089] In the formula, Indicates the total execution cost. The mathematical expression representing budget constraints and selection variable constraints is:

[0090]

[0091]

[0092] in Indicates the number of optional configurations. Indicates the selection of the first Indicator variables for various types of serverless functions Indicates the selection of the first This memory type is used for the indicator variable of the Reducer. and Use binary variables to ensure the uniqueness of the schedule.

[0093] Step 4: Evaluate the fitness of each solution in the current population. The mathematical expression for the objective function that minimizes the task completion time is:

[0094]

[0095] in Let represent the objective function that minimizes the task completion time. Indicates the total execution time. Indicates the execution time of the Map phase. Indicates the execution time of the coordinator phase. Indicates the execution time of the Reduce phase;

[0096] The mathematical expression for the objective function that minimizes execution cost is:

[0097]

[0098] in This represents the objective function that minimizes the execution cost. Indicates the total execution cost. Indicates the cost of serverless function calls. Indicates the execution cost of a serverless function. Indicates the request cost of the object storage system. The object storage system storage cost represents intermediate data. The serverless function call cost refers to the fixed cost incurred when calling a serverless function. The serverless function execution cost refers to the cost calculated based on the execution time and memory configuration during the execution of a serverless function. The object storage system refers to a distributed storage system used to store intermediate data. The object storage system request cost refers to the request cost incurred when accessing the object storage system. The object storage system storage cost refers to the storage cost of intermediate data in the object storage system.

[0099] Step 5: Perform non-dominated sorting on the current population, initialize the dominance count based on dominance relationships, iteratively construct non-dominated fronts, and calculate the crowding distance of solutions in each non-dominated front. Specifically, the non-dominated sorting method is as follows: initialize the list of dominant solutions and the dominance count for each solution, compare any combination of two solutions in the current population, and if the solution... It is superior to or equal to the solution in both the objective function of minimizing task completion time and the objective function of minimizing execution cost. It outperforms at least one of the objective functions: minimizing task completion time and minimizing execution cost. ,but Dominate ,Will join in The list of dominant solutions will Increment the dominance count by 1, and add the solution with a dominance count of 0 to the first non-dominated frontier. A dominance count of 0 indicates that the solution is not dominated by any other solution. Iterate through the next non-dominated frontier. The first non-dominated frontier has the lowest level, and the level of the next non-dominated frontier increases. Traverse all solutions dominated by each solution in the current non-dominated frontier, and decrement the dominance count of each traversed solution by 1. When the dominance count of a traversed solution drops to 0, the traversed solution is added to the next non-dominated frontier.

[0100] The method for calculating the congestion distance is to sort the solutions within each non-dominated frontier in ascending order according to the objective function value that minimizes the task completion time, thus obtaining a sequence. The sequence is obtained by sorting the values ​​of the objective function that minimize execution cost in ascending order. Will The crowding distance of the solution that minimizes or maximizes the objective function value is set to infinity. The crowding distance of the solution that minimizes or maximizes the objective function value is set to infinity; for any other solution in this non-dominated front... Its crowding distance is defined as: in Winning The normalized distance between two adjacent solutions minimizes the objective function of task completion time, and in Winning The normalized distance between two adjacent solutions is calculated to minimize the execution cost objective function; the sum of the two is... Crowded distances.

[0101] Step 6: Execute tournament selection, adaptive crossover strategy, and topology repair mutation strategy sequentially to generate offspring population. Specifically, tournament selection randomly selects a preset number of solutions from the current population for comparison, and selects solutions with lower rank of non-dominated front and larger crowding distance as parent solutions;

[0102] The adaptive crossover strategy employs a crossover method based on adaptive crossover point selection. At the selected crossover point, it swaps the node configurations of the two parent solutions and recalculates the total execution cost and total execution time of the solution. The formula for calculating the crossover point selection probability is as follows:

[0103]

[0104] In the formula, Indicates the selection of a location As the probability of choosing the intersection point For the intersection location index, For adaptive parameters, The length of the solution;

[0105] Topology repair mutation strategies include the following processes, such as Figure 3 As shown, the source node and target node exist Figure 3 The nodes correspond to the source node and the target node, respectively. , , , , , , , , , , The nodes represent the resource decision stages along the path, with each path corresponding to only one node at each resource decision stage. The topology repair mutation strategy includes the selection of mutated nodes, the determination of nodes, and the repair process, given a solution encoding sequence. = In the process of selecting mutation nodes, from a uniform distribution Randomly select a mutated node , This represents the total number of nodes in the decoded sequence.

[0106] Specifically, in the process of selecting mutation nodes, such as Figure 3 As shown, the initial path is Among the nodes This represents the node corresponding to the initial path during the Mapper memory configuration phase. This represents the node corresponding to the initial path during the Mapper count phase. This represents the node corresponding to the initial path in the stage of the Reducer's input object count. Selected as a mutated node. Figure 3 The dashed arrows in the path start from the nodes. Pointing to the node below The text is labeled "Select Mutant Nodes" to indicate the process of selecting mutated nodes. The nodes below... This refers to the selected mutated node.

[0107] In the process of determining nodes, such as Figure 3 As shown, from the mutated node A new edge is randomly selected from the adjacency list of the predecessor node to replace it. Figure 3 In the middle, node The node corresponding to the mutated node The predecessor node, Figure 3 The label "randomly select a new edge" indicates that an edge is randomly selected from multiple candidate edges. Figure 3 From the mutated node predecessor node There are three candidate edges pointing to the node. ,node ,node mutated nodes It will be replaced with any one of these three nodes, where the mutated node is... The selected path is indicated by a bold arrow. Edge replacement must satisfy the topological continuity constraint, that is, there exists an edge from the predecessor node to the target node of the new edge, and the target node of the new edge is not equal to the mutated node. The target node of the new edge represents the node that the new edge points to. Mutated node. It can mutate into a node. , , Any one of them, the selected path will mutate the node. Mutate into a node Each candidate mutated node has a corresponding subsequent path segment, which points to the node. ,node ,node The selected path starts from the node. Pointing to node , Figure 3 From the node To the node The path segment marked "connectivity check" indicates that a connectivity check is performed on that path segment.

[0108] During the repair process, such as Figure 3 As shown, the generated path undergoes constraint checks and repairs. From the node The path from the starting point to the destination node The path segment is marked "connectivity break," indicating that the new edge causes a mutated node in the decoded sequence. The following part is related to the target node. The connectivity break, that is, from the node To the target node The path is incomplete or does not exist. From node The path from the starting point to the destination node The path segment marked "Does not satisfy topological continuity constraint" indicates that the edge replacement violated the topological continuity constraint, meaning the newly selected edge does not meet the topological continuity requirement. From node The path from the starting point to the destination node The path segment marked "Reselect Edges to Rebuild Path" indicates that when a connectivity break or violation of topological continuity constraints is detected, a repair mechanism is triggered to check for a path from the disconnected node to the target node. If a path does not exist, identify and remove nodes that do not meet the resource constraints. Starting from the disconnected node, select edges that meet the resource constraints from the disconnected node's adjacency list to construct a path to the next node, and so on, until the target node is reached. .like Figure 4 As shown, all paths eventually converge, starting from node The path from the starting point to the target node From node The starting path also reaches the target node. This indicates that the repaired path may reach the target node through different resource decision-making stage nodes. .

[0109] Step 7: Periodically execute the heuristic search-guided solution injection strategy to inject the single-objective optimal solution into the offspring population. Specifically, every [percentage missing]... The offspring population is non-dominated and sorted. From the first non-dominated front, solutions are selected in descending order of crowding distance, and the solution with the largest crowding distance is chosen as the representative solution. The injection interval is set according to a preset heuristic search; the total execution cost or total execution time of representative solutions is extracted as a constraint; with the goal of minimizing execution cost, heuristic search is used to obtain the solution with the minimum execution time under the constraint of the total execution cost of representative solutions; with the goal of minimizing execution time, heuristic search is used to search for the solution with the minimum execution cost under the constraint of the total execution time of representative solutions; the generated single-objective optimal solution is injected into the offspring population to replace solutions with higher ranks or smaller crowding distances in the non-dominated sorting, or to supplement regions with smaller crowding distances in the offspring population, thus updating the offspring population.

[0110] Step 8: Merge the parent and offspring populations, and perform crowding distance selection on the merged temporary population to generate a new population. Specifically, the crowding distance selection method is as follows: perform non-dominated sorting on the temporary population, divide the temporary population into multiple non-dominated fronts, calculate the crowding distance of solutions for each non-dominated front, select solutions in ascending order of non-dominated front level, sort each non-dominated front by crowding distance from largest to smallest, and prioritize the selection of solutions with larger crowding distances, until the number of selected solutions reaches the preset population size, and generate a new population.

[0111] Step 9: Update the global Pareto optimal solution set. Specifically, when updating the global Pareto optimal solution set, merge the non-dominated front after the current iteration's non-dominated sorting with the historical optimal solution set and perform non-dominated sorting. Extract the solutions in the first non-dominated front after non-dominated sorting as the current global Pareto optimal solution set, and record the solution with the minimum execution cost and the solution with the minimum execution time in the current global Pareto optimal solution set.

[0112] Step 10: Determine if the time limit has been reached. If it has, output the current global Pareto optimal solution set and the total execution cost and total execution time corresponding to the current global Pareto optimal solution set. Otherwise, return to step 4 to continue iterating.

[0113] The technical effectiveness is verified through specific embodiments below. The experimental environment simulates the characteristics of a serverless platform, constructing 100 MapReduce job scheduling test instances, including 10 scheduling sizes (S1 to S10) and 10 task sizes (T1 to T10). Each combination of size and task size constitutes a test instance. The node processing rate is set according to the serverless platform. The experiment is repeated 15 times to reduce errors. The experimental platform is developed using Java SE 8, running on a Windows 10 (64-bit) operating system, with hardware configuration of an Intel® Core™ i7-12700 2.10 GHz CPU and 32 GB RAM. The evaluation metrics used are two multi-objective optimization standard indicators: hypervolume (HV) and inverse generation distance (IGD). A higher hypervolume indicates better solution set quality, while a lower inverse generation distance indicates better solution set convergence.

[0114] The heuristic search-guided non-dominated sorting genetic algorithm (ANSGA-II) of this invention was subjected to ablation experiments relative to the heuristic search-guided uninjection strategy, topology repair mutation strategy, and hybrid initialization strategy. Ablation variant 1 represents the algorithm variant of this invention that removes the heuristic search-guided uninjection strategy, ablation variant 2 represents the algorithm variant of this invention that removes the topology repair mutation strategy, and ablation variant 3 represents the algorithm variant of this invention that removes the hybrid initialization strategy. The hypervolume comparison of the ablation experiments is shown in Table 2.

[0115] Table 2 Comparison of overvolume ablation experiments

[0116]

[0117] Table 3 shows a comparison of reverse generational distances in ablation experiments:

[0118] Table 3 Comparison of reverse generational distances in ablation experiments

[0119]

[0120] The values ​​in Table 2 represent the average hypervolume of the ANSGA-II algorithm and three ablation variants after 15 repetitions at 10 different job scales. The values ​​in Table 3 represent the average reverse generation distance of the ANSGA-II algorithm and three ablation variants after 15 repetitions at 10 different job scales. Based on the data analysis in Tables 2 and 3, the ANSGA-II algorithm of this invention achieved optimal or near-optimal performance across all job scales. According to the data in Table 2, the hypervolume of the ANSGA-II algorithm ranges from 0.95012 to 0.99967, with an optimal value of 0.99967. According to the data in Table 3, the reverse generation distance of the ANSGA-II algorithm ranges from 0.02552 to 0.12962. All three ablation variants exhibited significant performance degradation. For example, in the S1 scale, according to Table 2, the hypervolume of ablation variant 1 was 0.93527, significantly lower than ANSGA-II's 0.95012; according to Table 3, the reverse generation distance of ablation variant 1 was 0.11059, significantly higher than ANSGA-II's 0.10385. In the S8 scale, according to Table 2, the hypervolume of ablation variant 2 was 0.93244, significantly lower than ANSGA-II's 0.97643; according to Table 3, the reverse generation distance of ablation variant 2 was 0.10992, significantly higher than ANSGA-II's 0.04798. In the S9 scale, according to Table 3, the reverse generation distance of ablation variant 2 reached 0.15708, significantly higher than ANSGA-II's 0.08980. These results demonstrate that removing any of the three strategies—heuristic search-guided solution injection, topology repair mutation, and hybrid initialization—leads to a significant decrease in algorithm performance. All three strategies play crucial roles in improving algorithm performance. Specifically, the heuristic search-guided solution injection and hybrid initialization strategies are key to enhancing algorithm stability, while the topology repair mutation strategy is essential for maintaining the feasibility and convergence of solutions. This invention combines the heuristic search-guided solution injection, topology repair mutation, and hybrid initialization strategies to propose a heuristic search-guided non-dominated sorting genetic algorithm (ANSGA-II), which solves the complex problem of solving serverless MapReduce multi-objective scheduling within a finite time limit.

[0121] By comparing the heuristic search-guided non-dominated sorting genetic algorithm (ANSGA-II) of this invention with four existing comparative algorithms, the advantages of this invention in solution set quality and convergence performance in serverless MapReduce multi-objective scheduling scenarios are verified. The four existing comparative algorithms are: decomposition-based multi-objective evolutionary algorithm (MOEA / D), multi-objective gray wolf optimization algorithm (MOGWO), multi-objective particle swarm optimization algorithm (MOPSO), and non-dominated sorting genetic algorithm (NSGA-II). Specific comparative experimental results are shown in Tables 4, 5, 6, and 7. Figure 5 , Figure 6 , Figure 7 , Figure 4 The comparison of the hypervolume of this invention with four comparison algorithms at different scales is shown in Table 4:

[0122] Table 4. Hypervolume comparison between the present invention and four comparative algorithms at different scales.

[0123]

[0124] As can be seen from Table 4, in all 10 job scale test instances, the supervolume of the ANSGA-II of the present invention achieved the highest value relative to the other four comparison algorithms, and the value range was concentrated between 0.9972 and 0.9998. Compared with the four comparison algorithms, there were no obvious degradation points. This shows that the ANSGA-II of the present invention can continuously obtain a Pareto solution set with a more uniform distribution and a larger coverage under different job scales.

[0125] The inverse generation distance comparison between this invention and four comparison algorithms at different scales is shown in Table 5:

[0126] Table 5. Comparison of reverse generation distance between the present invention and four comparison algorithms at different scales.

[0127]

[0128] As shown in Table 5, in 10 job scale test instances, the ANSGA-II of the present invention consistently achieved the minimum reverse generation distance index compared to the other four comparison algorithms, and the value remained below 0.0033. In contrast, the reverse generation distance of the other four comparison algorithms increased significantly at some scales. This indicates that the ANSGA-II of the present invention is more likely to approximate the true Pareto front at different job scales and has better global convergence ability.

[0129] The supervolume comparison of this invention with four comparative algorithms under different task sizes is shown in Table 6:

[0130] Table 6. Hypervolume Comparison of the Invention and Four Comparison Algorithms under Different Task Sizes.

[0131]

[0132] As can be seen from the values ​​in Table 6, as the task size gradually increases from T1 to T10, the supervolume of the ANSGA-II of the present invention remains above 0.9974 under all task sizes and shows a slow upward trend overall. In contrast, the other four comparison algorithms show obvious deterioration when the task size is small, indicating that the present invention has better adaptability and scalability to different task sizes.

[0133] The inverse generation distance comparison between this invention and four comparison algorithms under different task sizes is shown in Table 7:

[0134] Table 7. Comparison of reverse generation distance between the present invention and four comparison algorithms under different task sizes.

[0135]

[0136] As shown in Table 7, in 10 test cases with different task sizes, the ANSGA-II of the present invention consistently outperforms the other four comparative algorithms in terms of the reverse generation distance index. Moreover, the reverse generation distance is less than 0.0031 for all task sizes from T1 to T10, while the other four comparative algorithms show a sudden increase in value for several task sizes. This indicates that the ANSGA-II of the present invention can still maintain stable convergence performance and solution set accuracy when the task size changes significantly.

[0137] Figure 4 The horizontal axis represents the job size, including 10 job size levels from S1 to S10, and the vertical axis represents the hypervolume; a higher value indicates better solution set quality. Figure 5 As can be seen, the hypervolume curve of this invention consistently ranks higher than the other four comparative algorithms and remains stable across different job scales, with the hypervolume consistently above 0.997, reaching a maximum of 0.9998. This demonstrates that ANSGA-II of this invention can obtain high-quality Pareto optimal solution sets under different job scales. In contrast, the hypervolume curves of the other four comparative algorithms, MOEA / D, MOGWO, MOPSO, and NSGA-II, fluctuate significantly and are considerably lower than those of ANSGA-II. Particularly in the job scale S8 test instance, MOPSO's hypervolume is only 0.9613, significantly lower than the 0.9998 of this invention, a difference of 3.85 percentage points, fully demonstrating the significant advantage of this invention in solution set quality.

[0138] Figure 5 The horizontal axis represents the job size, including 10 job size levels from S1 to S10, and the vertical axis represents the reverse generation distance. The lower the value, the better the convergence of the solution set.Figure 5 The graph shows a line graph illustrating the reverse generation distance between the present invention and four comparison algorithms as a function of job size. Figure 6 It can be seen that the reverse generation distance curve of the present invention is always below the other four comparison algorithms, and the curve is smooth and stable with small fluctuations. As the job scale increases from S1 to S10, the reverse generation distance curve of the present invention ANSGA-II remains basically horizontal with no obvious upward or downward trend. This shows that the present invention ANSGA-II can quickly converge to the Pareto optimal solution set under different job scales.

[0139] Figure 6 The horizontal axis represents the task size, including 10 task size levels from T1 to T10, and the vertical axis represents the hypervolume; a higher value indicates a better solution set quality. Figure 6 The graph shows the supervolume of this invention and four comparative algorithms as a function of task size. Figure 7 As can be seen, the hypervolume curve of this invention exhibits a stable upward trend. As the task size gradually increases from T1 to T10, the hypervolume curve of this invention steadily rises from 0.9974 to 0.9997, showing an overall monotonically increasing trend. It maintains a high level across different task sizes, with the hypervolume value consistently above 0.997, reaching a maximum of 0.9997. This indicates that this invention has good adaptability to different task sizes, can stably obtain high-quality Pareto optimal solution sets, and further improves the quality of the solution set as the task size increases. In contrast, the hypervolume curves of the other four comparison algorithms fluctuate significantly. The curves of MOEA / D and NSGA-II show a slow upward trend, but with a low starting point and obvious fluctuations. The curves of MOGWO and MOPSO fluctuate the most dramatically. Although the hypervolume of the comparison algorithms improves with increasing task size, it is still significantly lower than that of this invention, fully demonstrating the stability and superiority of this invention in terms of solution set quality.

[0140] Figure 7 The horizontal axis represents the task size, including 10 task size levels from T1 to T10, and the vertical axis represents the backward generation distance. The lower the value, the better the convergence of the solution set. Figure 7 The graph shows a line graph illustrating the reverse generation distance between the present invention and four comparison algorithms as a function of task size. Figure 4As can be seen, the backward generation distance curve of ANSGA-II of this invention is consistently lower than that of the other four comparative algorithms, and the curve is smooth and stable. As the task size gradually increases from T1 to T10, the backward generation distance curve of ANSGA-II of this invention fluctuates slightly between 0.0006 and 0.0031, maintaining a low level overall without a significant upward or downward trend. This indicates that this invention can quickly converge to the Pareto optimal solution set under different task sizes. In contrast, the backward generation distance curves of the other four comparative algorithms fluctuate more. The curves of MOEA / D and NSGA-II show a trend of first decreasing and then fluctuating. The curve of MOGWO fluctuates significantly within the range of T1 to T10. The curve of MOPSO fluctuates the most dramatically. As the task size increases, although the backward generation distance of the four comparative algorithms decreases, it is still significantly higher than that of ANSGA-II, and the fluctuation amplitude is large, fully demonstrating the significant advantages of ANSGA-II of this invention in terms of convergence speed and stability.

[0141] In summary, according to Table 4, Figure 5 and Table 5, Figure 4 Based on the data, under different operational scale test instances, the ANSGA-II of this invention outperforms the other four comparative algorithms in both hypervolume and reverse generation distance metrics. In the operational scale S1 test instance, according to Table 4 and... Figure 5 According to the data, the hypervolume of ANSGA-II is 0.9993, which is higher than 0.9935 for the MOEA / D algorithm and 0.9811 for MOGWO; based on Table 5 and Figure 4 According to the data, the reverse generation distance of ANSGA-II is 0.0007, lower than MOEA / D's 0.0232 and MOGWO's 0.0269. In the S8 test instance, based on Table 4 and... Figure 5 The data shows that the hypervolume of ANSGA-II is 0.9998, MOEA / D is 0.9940, and MOPSO is 0.9613; according to Table 5 and Figure 6 According to the data, the reverse generation distance of ANSGA-II is 0.0013, MOEA / D is 0.0220, and MOPSO is 0.0505. Based on Table 6... Figure 7 and Table 7, Figure 6 Based on the data, under test instances of different task sizes, the ANSGA-II of this invention exhibits good robustness in both hypervolume and reverse generation distance metrics. According to Table 6 and... Figure 7 According to the data, the hypervolume of the ANSGA-II of this invention remained above 0.997 in all test cases, ranging from 0.9974 to 0.9997; based on Table 7 and Figure 6Based on the data, the reverse generation distance values ​​of ANSGA-II in this invention are distributed in the range of 0.0006 to 0.0031. Therefore, under different task sizes, the algorithm of this invention can converge to the Pareto optimal solution set. In contrast, the other four comparative algorithms have greater volatility. MOEA / D, according to Table 6 and Figure 7 The data shows that its supervolume fluctuates between 0.9898 and 0.9973; according to Table 7 and ​ The data shows that the reverse generation distance value fluctuates between 0.0133 and 0.0371. Therefore, experimental data demonstrates that, compared with the other four comparison algorithms, the ANSGA-II of this invention improves the solution set quality and convergence of serverless MapReduce multi-objective scheduling, exhibits better stability and robustness, and is better adapted to the characteristics of serverless environments.

Claims

1. A serverless MapReduce multi-objective scheduling optimization method, characterized in that, The method comprises the following steps: Step 1, initializing the operation parameters of the heuristic search guided non-dominated sorting genetic algorithm, including population size, crossover rate, mutation rate and heuristic search guidance injection interval; Step 2, constructing a directed acyclic graph of resource allocation of the serverless MapReduce job, and encoding the resource allocation decision as a solution from a source node to a target node; Step 3, generating an initial population by using a hybrid initialization strategy; Step 4, evaluating the fitness of each solution in the current population by using a target function for minimizing task completion time and a target function for minimizing execution cost, and taking the current population as a parent population; Step 5, non-dominantly sorting the current population, initializing the dominance count based on the dominance relationship, iteratively constructing a non-dominated front, and calculating the crowding distance of the solutions in each non-dominated front; Step 6, sequentially performing tournament selection, an adaptive crossover strategy and a topology repair mutation strategy to generate a child population; Step 7, periodically performing a heuristic search guided solution injection strategy, injecting a single-objective optimal solution into the child population, and updating the child population; Step 8, merging the parent population and the child population, and performing crowding distance selection on the merged temporary population to generate a new population; Step 9, updating the global Pareto optimal solution set; Step 10, judging whether the time limit is reached, if yes, outputting the current global Pareto optimal solution set and the total execution cost and total execution time corresponding to the current global Pareto optimal solution set, and otherwise returning to step 4 for iteration. 2.The serverless MapReduce multi-objective scheduling optimization method of claim 1, wherein, In step 2, the method for constructing the directed acyclic graph model is to encode the resource allocation decision as a solution structure, and model each solution as a source node in the directed acyclic graph. To the target node The path, Indicates the source node, The target node is represented by [target node], and the other nodes in the directed acyclic graph are [other nodes]. It means that, among them, This represents the stage index of the resource decision-making phase. This represents the node index within the resource decision-making phase. Indicates the first Phase 1 1 node The values ​​are 1, 2, 3, 4, and 5. A value of 1 corresponds to the Mapper memory configuration phase; A value of 2 corresponds to the number of Mappers in the current stage. When the value is 3, it corresponds to the stage of the number of input objects for the Reducer; When the value is 4, it corresponds to the coordinator memory configuration stage; When the value is 5, it corresponds to the Reducer memory configuration stage; decoding sequence. = , These represent the nodes corresponding to each resource decision stage of the path, with each path corresponding to only one node in each stage. Each edge in the directed acyclic graph connects two nodes in adjacent stages, representing the transition from resource configuration selection in one stage to selection in another. Each edge stores the execution time and cost of the corresponding resource configuration. The validity of a solution is determined based on the fact that all adjacent nodes are connected by edges in the directed acyclic graph, and the cumulative execution time and cost do not exceed a preset threshold. The cumulative execution cost represents the time and cost from the source node during the path search process. The cumulative execution time is the sum of the execution costs of all edges on the path to the current node, representing the execution time from the source node during the path search process. The sum of execution times for all edges on the path to the current node. When the path search is complete, i.e., the path originates from the source node... Reach the target node At that time, the cumulative execution cost equals the total execution cost, and the cumulative execution time equals the total execution time. 3.The serverless MapReduce multi-objective scheduling optimization method of claim 2, wherein, In step 3, the hybrid initialization strategy comprises a heuristic search guidance process and a constraint-aware random generation process, in the heuristic search guidance process, a solution with minimum execution cost and a solution with minimum execution time are generated by using a heuristic search, and in the constraint-aware random generation process, different solutions are constructed by using a depth-first search and a dynamic pruning method, wherein, When using heuristic search to generate the solution with minimum execution cost, the adaptive threshold method is adopted. The implementation process is as follows: starting from the source node An open list and a closed list are maintained, the open list stores nodes to be expanded, and the closed list stores expanded nodes. Each time, the node with the minimum evaluation function value is selected from the open list for expansion, and the evaluation function is defined as wherein represents a node of an evaluation function value, represents a node in a directed acyclic graph, represents an actual execution cost from a source node to a node , represents an estimated execution cost from a node to a target node , is a dynamically adjusted relaxation factor, and when the expanded node is the target node , the backtracked path results in a solution of minimum execution cost. When using heuristic search to generate the solution with minimum execution time, the reverse search is adopted, and the implementation process is as follows: from the target node to the source node , the reverse search is performed, an open list and a closed list are maintained, each time the node with the minimum cumulative execution time is selected from the open list for expansion, and when the expanded node is the source node , the path is backtracked to obtain the solution with the minimum execution time; The specific process of the depth first search is as follows: initializing the current node as the source node , initializing the path as empty, adding the source node to the path, obtaining the adjacency list of the current node if the current node is not equal to the target node , terminating the generation of the current path if the adjacency list is empty, and randomly and uniformly selecting an edge from the adjacency list, updating the target node connected by the selected edge as the current node, and adding the current node, the execution cost of the edge, and the execution time to the path. ​ The specific process of the dynamic pruning method is as follows: after each node expansion, the accumulated execution cost and accumulated execution time from the source node to the current node are calculated, when the accumulated execution cost is greater than the budget limit or the accumulated execution time is greater than the deadline, the expansion of the current path is terminated, the last node in the path that satisfies the resource constraint condition is traced back to, the attempted edge is removed from the adjacency list of the last node that satisfies the resource constraint condition, a satisfied resource constraint condition edge is randomly selected for expansion again, if the edge that satisfies the resource constraint condition cannot be found after backtracking, the current path is abandoned and a new path is generated from the source node again. ​​​​ 4. The serverless MapReduce multi-objective scheduling optimization method of claim 3, wherein, the resource constraint conditions comprise function concurrency limitation, temporary storage limitation, quality of service constraint and selection variable constraint, wherein, the mathematical expression of the function concurrency limitation is wherein, represents the number of serverless functions used in the Map phase, represents the number of steps of the Reduce phase, represents the number of Reducers in the Reduce phase step , and represents the maximum concurrent resource limit allowed by the serverless platform; the mathematical expression of the temporary storage limitation is In the formula, represents the index of the Map task, represents the intermediate result size of the th Map task, in MB, represents the platform limit; the quality of service constraint comprises task completion time constraint and execution cost constraint, the mathematical expression of the task completion time constraint is In the formula, denotes the total execution time, denotes the deadline; the mathematical expression of the execution cost constraint is wherein denotes the total execution cost, denotes the budget limit, the mathematical expression of the selection variable constraint is wherein represents the number of optional configurations, represents an indicator variable for selecting the type of serverless function, represents an indicator variable for selecting the type of memory for the Reducer, the indicator variable and are binary variables ensuring the uniqueness of the schedule.

5. The serverless MapReduce multi-objective scheduling optimization method of claim 4, wherein, in step 4, the mathematical expression of the target function for minimizing task completion time is wherein represents an objective function to minimize the task completion time, represents the total execution time, represents the Map phase execution time, represents the Coordinator phase execution time, represents the Reduce phase execution time; the mathematical expression of the target function for minimizing execution cost is wherein represents a target function of minimizing execution cost, represents total execution cost, represents serverless function invocation cost, represents serverless function execution cost, represents object storage system request cost, represents object storage system storage cost of intermediate data, serverless function invocation cost refers to a fixed fee generated when a serverless function is invoked, serverless function execution cost refers to a fee calculated according to execution time and memory configuration during execution of a serverless function, object storage system refers to a distributed storage system used to store intermediate data, object storage system request cost refers to a request fee generated when an object storage system is accessed, and object storage system storage cost refers to a storage fee of intermediate data in an object storage system.

6. The serverless MapReduce multi-objective scheduling optimization method of claim 5, wherein, In step 5, the non-dominated sorting method is that, initializing a dominated solution list and a dominated count of each solution, comparing combinations of any two solutions in the current population, if a solution is better than or equal to a solution in both the objective function of minimizing the task completion time and the objective function of minimizing the execution cost, and is better than a solution in at least one of the objective function of minimizing the task completion time and the objective function of minimizing the execution cost, then dominates , the solution is added to the dominated solution list of the solution , the dominated count of the solution is added by 1, the solution with the dominated count of 0 is released into the first non-dominated front, the dominated count of 0 means that the solution is not dominated by any other solution, the next non-dominated front is iteratively calculated, all solutions dominated by each solution in the current non-dominated front are traversed, the dominated count of each traversed solution is reduced by 1, when the dominated count of the traversed solution is reduced to 0, the traversed solution is added into the next non-dominated front; wherein, the first non-dominated front has the lowest level, and the level of the next non-dominated front is increased. the calculation method of the crowding distance is that, for each non-dominated front, the solutions are sorted according to the values of the target function for minimizing task completion time and the target function for minimizing execution cost, and the sorting sequence is obtained, the crowding distance of the solution with the minimum or maximum value in the non-dominated front in the target function for minimizing task completion time is set to infinity, the crowding distance of the solution with the minimum or maximum value in the non-dominated front in the target function for minimizing execution cost is set to infinity, and the crowding distance of other solutions in the non-dominated front is the sum of the normalized distances of the adjacent solutions in the sorting sequence in the target function for minimizing task completion time and the target function for minimizing execution cost.

7. The serverless MapReduce multi-objective scheduling optimization method of claim 6, wherein, In step 6, the tournament selection randomly selects a preset number of solutions from the current population for comparison, and selects solutions with lower rank and larger crowding distance in the non-dominated front as parent solutions; The adaptive crossover strategy adopts a crossover method based on adaptive crossover point selection, exchanges the node configurations of two parent solutions at the selected crossover point position, and recalculates the total execution cost and total execution time of the solution, wherein the calculation formula of the crossover point selection probability is wherein denotes the selection position as the intersection selection probability, is the intersection position index, is the adaptive parameter, is the length of the solution; The topology repairing variation strategy includes a variation node selection process, a node determination process, and a repairing process, and is directed to a decoding sequence = In the variation node selection process, a variation node is randomly selected from a uniform distribution , , wherein N represents the total number of nodes in the decoding sequence.​ In the process of determining nodes, from mutated nodes A new edge is randomly selected from the adjacency list of the predecessor node to replace it. The edge replacement must satisfy the topological continuity constraint, that is, there exists an edge from the predecessor node to the target node of the new edge, and the target node of the new edge is not equal to the mutated node. The target node of the new edge represents the node that the new edge points to; during the repair process, if the new edge causes a mutated node in the decoded sequence... The following part is related to the target node. If connectivity breaks or topological continuity constraints are violated, a repair mechanism is triggered to check for a connection from the disconnected node to the target node. If a path does not exist, identify and remove nodes that do not meet the resource constraints. Starting from the disconnected node, select edges that meet the resource constraints from the disconnected node's adjacency list to construct a path to the next node, and so on, until the target node is reached. .

8. The serverless MapReduce multi-objective scheduling optimization method of claim 7, wherein, In step 7, the specific process of the heuristic search guided solution injection strategy is to inject a solution every The offspring population is non-dominantly sorted, and the solutions are sorted in descending order of crowding distance from the first non-dominated front, and the solution with the largest crowding distance is selected as the representative solution, The injection interval of the preset heuristic search guided injection is extracted; and the total execution cost or total execution time of the representative solution is taken as a constraint condition; With the objective of minimizing the execution cost, under the constraint of the total execution cost of the representative solution, a heuristic search is used to obtain a solution with minimum execution time; With the objective of minimizing the execution time, under the constraint of the total execution time of the representative solution, a heuristic search is used to search for a solution with minimum execution cost; the generated single-objective optimal solution is injected into the child population to replace solutions with higher non-dominated rank or smaller crowding distance, or to supplement the smaller crowded distance area in the child population, and the child population is updated.

9. The serverless MapReduce multi-objective scheduling optimization method of claim 8, wherein, In step 8, the crowding distance selection method is as follows: non-dominated sorting is performed on the temporary population, the temporary population is divided into multiple non-dominated fronts, the crowding distance of the solutions in each non-dominated front is calculated, the solutions are selected in order of the rank of the non-dominated front from low to high, and for each non-dominated front, the solutions are sorted in order of the crowding distance from large to small, and the solutions with larger crowding distance are preferentially selected, until the number of selected solutions reaches the preset population size, and a new population is generated.

10. The serverless MapReduce multi-objective scheduling optimization method of claim 9, wherein, In step 9, when updating the global Pareto optimal solution set, the non-dominated fronts after non-dominated sorting of the current iteration and the historical optimal solution set are merged and non-dominated sorted, and the solutions in the first non-dominated front after non-dominated sorting are extracted as the current global Pareto optimal solution set, and the solution with minimum execution cost and the solution with minimum execution time in the current global Pareto optimal solution set are recorded.

Images