Serverless mapreduce multi-objective scheduling optimization method

Abstract

The present application relates to cloud computing and distributed computing scheduling technical field, specifically belong to a kind of serverless MapReduce multi-objective scheduling optimization method, including: constructing directed acyclic graph, resource allocation decision is encoded as the solution from source node to target node, initial population is generated using hybrid initialization strategy, fitness evaluation, the crowded distance of solution in each non-dominated front is calculated, in turn execute tournament selection, adaptive crossover strategy, topological repair mutation strategy, periodically execute heuristic search guided solution injection strategy, crowded distance selection is carried out, the global Pareto optimal solution set is updated, the global Pareto optimal solution set of current time output is achieved, otherwise return iteration. The present application has the positive effect of balancing the execution cost and execution time of MapReduce job, improving the global Pareto optimal solution set quality of multi-objective scheduling scheme, enhancing the convergence speed and robustness of algorithm.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and distributed computing scheduling technology, specifically an optimization method for multi-objective resource scheduling of MapReduce data-intensive workflows in a serverless computing architecture. Background Technology

[0002] Serverless computing is a key paradigm in cloud computing development, offering users core advantages such as automatic scaling, pay-as-you-go billing, and maintenance-free operation through Function as a Service (Quadratye). With the continuous growth in big data processing demands, serverless platforms have begun deploying the MapReduce distributed computing framework. In serverless MapReduce jobs, the Mapper (task processing unit) of the Map phase and the Reducer (task processing unit) of the Reduce phase are implemented as serverless functions. The execution flow of a serverless MapReduce job includes multiple Mapper functions, a coordinator function, and multiple Reducer functions. In traditional cloud computing environments, research on MapReduce scheduling mainly focuses on single-objective optimization, such as minimizing execution time or reducing execution cost. In practical applications, scheduling systems need to consider multiple conflicting objectives simultaneously, including resource efficiency, execution time, energy consumption, and operating costs. Multi-objective optimization algorithms provide direction for solving multi-objective conflict problems; representative algorithms include non-dominated sorting genetic algorithms, decomposition-based multi-objective evolutionary algorithms, multi-objective particle swarm optimization algorithms, and multi-objective gray wolf optimization algorithms.

[0003] However, serverless MapReduce job scheduling faces the following technical drawbacks: First, there is a multi-objective conflict between the execution cost and execution time of MapReduce jobs. Under the premise of satisfying resource constraints, pursuing lower execution cost requires reducing the number of functions or choosing a smaller memory configuration, which increases execution time. Conversely, pursuing lower execution time requires more concurrent functions or higher resource configuration, which increases execution cost. Second, traditional multi-objective evolutionary algorithms struggle to converge to the optimal Pareto front within a limited execution time when dealing with serverless MapReduce job scheduling problems, and are prone to generating solutions that do not satisfy resource constraints during the evolutionary search process. Third, while heuristic search can quickly find the optimal solution, it is only applicable to single-objective optimization scenarios and cannot directly handle multi-objective optimization problems. Therefore, designing a multi-objective optimization framework that accelerates convergence and improves the quality of the global Pareto optimal solution set while ensuring that the solution satisfies resource constraints is a problem that needs to be solved in the optimization of MapReduce job scheduling in serverless environments. Summary of the Invention

[0004] The present invention provides a serverless MapReduce multi-objective scheduling optimization method, which solves the technical problems of multi-objective conflict between execution cost and execution time in the MapReduce job scheduling process in serverless environment, slow convergence speed of traditional multi-objective evolutionary algorithms and easy generation of solutions that do not meet resource constraints, so as to achieve the goal of balancing the execution cost and execution time of MapReduce jobs, improving the algorithm convergence speed and the uniformity of the global Pareto optimal solution set distribution.

[0005] The present invention provides a serverless MapReduce multi-objective scheduling optimization method, characterized by comprising the following steps:

[0006] Step 1: Initialize the running parameters of the heuristic search-guided non-dominated sorting genetic algorithm, including population size, crossover rate, mutation rate, and heuristic search-guided injection interval;

[0007] Step 2: Construct a directed acyclic graph for resource allocation of serverless MapReduce jobs, and encode the resource allocation decision as a solution from the source node to the target node;

[0008] Step 3: Generate the initial population using a hybrid initialization strategy;

[0009] Step 4: Evaluate the fitness of each solution in the current population. Use the objective function of minimizing task completion time and the objective function of minimizing execution cost to evaluate each solution, and use the current population as the parent population.

[0010] Step 5: Sort the current population by non-dominated ordering, initialize the dominance count based on the dominance relationship, iteratively construct the non-dominated front, and calculate the crowding distance of the solution in each non-dominated front;

[0011] Step 6: Perform tournament selection, adaptive crossover strategy, topology repair mutation strategy in sequence and generate offspring population;

[0012] Step 7: Periodically execute the solution injection strategy guided by heuristic search to inject the single-objective optimal solution into the offspring population and update the offspring population;

[0013] Step 8: Merge the parent and offspring populations, perform crowding distance selection on the merged temporary population, and generate a new population;

[0014] Step 9: Update the global Pareto optimal solution set;

[0015] Step 10: Determine if the time limit has been reached. If it has, output the current global Pareto optimal solution set and the total execution cost and total execution time corresponding to the current global Pareto optimal solution set. Otherwise, return to step 4 to continue iterating.

[0016] Furthermore, in step 2, the method for constructing the directed acyclic graph model is to encode the resource allocation decision as a solution structure, and model each solution as a path from the source node in the directed acyclic graph. To the target node The path, Indicates the source node, The target node is represented by [target node], and the other nodes in the directed acyclic graph are [other nodes]. It means that, among them, This represents the stage index of the resource decision-making phase. This represents the node index within the resource decision-making phase. Indicates the first Phase 1 1 node The values ​​are 1, 2, 3, 4, and 5. A value of 1 corresponds to the Mapper memory configuration phase; A value of 2 corresponds to the number of Mappers in the current stage. When the value is 3, it corresponds to the stage of the number of input objects for the Reducer; When the value is 4, it corresponds to the coordinator memory configuration stage; When the value is 5, it corresponds to the Reducer memory configuration stage; decoding sequence. = , These represent the nodes corresponding to each resource decision stage of the path, with each path corresponding to only one node in each resource decision stage. Each edge in the directed acyclic graph connects two nodes in adjacent stages, representing the transition from resource configuration selection in one stage to selection in another. Each edge stores the execution time and cost of the corresponding resource configuration. The validity of a solution is determined based on the fact that all adjacent nodes are connected by edges in the directed acyclic graph, and the cumulative execution time and cumulative execution cost do not exceed a preset threshold. Here, the cumulative execution cost represents the time and cost from the source node during the path search process. The cumulative execution time is the sum of the execution costs of all edges on the path to the current node, representing the execution time from the source node during the path search process. The sum of execution times for all edges on the path to the current node. When the path search is complete, i.e., the path originates from the source node... Reach the target node At that time, the cumulative execution cost equals the total execution cost, and the cumulative execution time equals the total execution time.

[0017] Furthermore, in step 3, the hybrid initialization strategy includes a heuristic search-guided process and a constraint-aware random generation process. In the heuristic search-guided process, heuristic search is used to generate solutions with minimum execution cost and minimum execution time. In the constraint-aware random generation process, different solutions are constructed through depth-first search and dynamic pruning methods.

[0018] When using heuristic search to generate the solution with minimum execution cost, an adaptive threshold approach is employed. The process involves starting from the source node... Initially, maintain an open list and a closed list. The open list stores nodes to be expanded, and the closed list stores expanded nodes. Each time, select the node with the smallest evaluation function value from the open list for expansion. The evaluation function is defined as follows:

[0019]

[0020] in Represents a node The evaluation function value, This represents a node in a directed acyclic graph. Indicates from the source node To the node The actual implementation cost Indicates from node To the target node The estimated execution cost, The relaxation factor is dynamically adjusted when the expanded node is the target node. When the backtracking path yields the solution with the minimum execution cost;

[0021] When using heuristic search to generate a solution with the minimum execution time, a reverse search approach is employed. The process involves starting from the target node... To the source node Perform a reverse search, maintaining an open list and a closed list. Each time, select the node with the shortest cumulative execution time from the open list for expansion. When the expanded node is the source node... When the time is right, the backtracking path yields the solution with the minimum execution time;

[0022] The specific process of depth-first search is as follows: starting from the source node... Begin by initializing the current node as the source node. The initial path is empty, and the source node is set. Add a path when the current node is not equal to the target node. When the current node is empty, obtain the adjacency list of the current node. The adjacency list stores all edges originating from the current node. If the adjacency list is empty, terminate the generation of the current path. Randomly and evenly select an edge from the adjacency list, update the target node connected to the selected edge to the current node, and add the execution cost and execution time of the current node and edge to the path.

[0023] The specific process of the dynamic pruning method is as follows: after each node expansion, calculate the number of nodes from the source node. The cumulative execution cost and cumulative execution time up to the current node, when the cumulative execution cost exceeds the budget limit. Or the cumulative execution time exceeds the deadline. If the current path cannot be expanded, backtrack to the last node in the path that satisfies the resource constraints. Remove the tried edges from the adjacency list of the last node that satisfies the resource constraints, and randomly select an edge that satisfies the resource constraints again for expansion. If no edge that satisfies the resource constraints can be found after backtracking, abandon the current path and start again from the source node. Start generating a new path.

[0024] Furthermore, resource constraints include function concurrency limits, temporary storage limits, quality of service constraints, and selection variable constraints, among which,

[0025] The mathematical expression for the function concurrency limit is:

[0026]

[0027] In the formula, This indicates the number of serverless functions used in the Map phase. This indicates the number of steps in the Reduce phase. Indicates the Reduce phase. The number of Reducers in each step This indicates the maximum concurrent resource limit allowed by the serverless platform;

[0028] The mathematical expression for the temporary storage limit is:

[0029]

[0030] In the formula, Indicates the index of the Map task. Indicates the first The size of intermediate results for each Map task, in MB. This indicates platform limitations;

[0031] Service quality constraints include task completion time constraints and execution cost constraints. The mathematical expression for the task completion time constraint is:

[0032]

[0033] In the formula, Indicates the total execution time. Indicates the deadline;

[0034] The mathematical expression for the execution cost constraint is:

[0035]

[0036] In the formula, Indicates the total execution cost. The mathematical expression representing budget constraints and selection variable constraints is:

[0037]

[0038]

[0039] in Indicates the number of optional configurations. Indicates the selection of the first Indicator variables for various types of serverless functions Indicates selecting the first This memory type is used for the indicator variable of the Reducer. and Use binary variables to ensure the uniqueness of the schedule.

[0040] Furthermore, in step 4, the mathematical expression for the objective function that minimizes the task completion time is:

[0041]

[0042] in Let represent the objective function that minimizes the task completion time. Indicates the total execution time. Indicates the execution time of the Map phase. Indicates the execution time of the coordinator phase. Indicates the execution time of the Reduce phase;

[0043] The mathematical expression for the objective function that minimizes execution cost is:

[0044]

[0045] in This represents the objective function that minimizes the execution cost. Indicates the total execution cost. Indicates the cost of serverless function calls. Indicates the execution cost of a serverless function. Indicates the request cost of the object storage system. The object storage system storage cost represents intermediate data. The serverless function call cost refers to the fixed cost incurred when calling a serverless function. The serverless function execution cost refers to the cost calculated based on the execution time and memory configuration during the execution of a serverless function. The object storage system refers to a distributed storage system used to store intermediate data. The object storage system request cost refers to the request cost incurred when accessing the object storage system. The object storage system storage cost refers to the storage cost of intermediate data in the object storage system.

[0046] Furthermore, in step 5, the non-dominated sorting method involves initializing the dominant solution list and dominance count for each solution, comparing any combination of two solutions in the current population, and if the solution... It is superior to or equal to the solution in both the objective function of minimizing task completion time and the objective function of minimizing execution cost. It outperforms at least one of the objective functions: minimizing task completion time and minimizing execution cost. ,but Dominate ,Will join in The list of dominant solutions will The dominance count is incremented by 1. Solutions with a dominance count of 0 are added to the first non-dominated frontier. A dominance count of 0 indicates that a solution is not dominated by any other solution. The next non-dominated frontier is calculated iteratively. All solutions dominated by each solution in the current non-dominated frontier are traversed. The dominance count of each traversed solution is decremented by 1. When the dominance count of a traversed solution drops to 0, the traversed solution is added to the next non-dominated frontier. The first non-dominated frontier has the lowest level, and the level of the next non-dominated frontier increases.

[0047] The crowding distance is calculated as follows: for each non-dominated frontier, sort the solutions according to the values ​​of the objective function that minimizes task completion time and the objective function that minimizes execution cost, and obtain a sorted sequence. Set the crowding distance of the solution with the smallest or largest value on the objective function that minimizes task completion time in the non-dominated frontier to infinity, and set the crowding distance of the solution with the smallest or largest value on the objective function that minimizes execution cost in the non-dominated frontier to infinity. The crowding distance of other solutions in the non-dominated frontier is the sum of the normalized distances of the solutions at adjacent positions in the sorted sequence on the objective functions that minimize task completion time and minimize execution cost.

[0048] Furthermore, in step 6, the tournament selects a preset number of solutions from the current population for comparison, and selects the solution with a lower level of non-dominant front and a larger crowding distance as the parent solution.

[0049] The adaptive crossover strategy employs a crossover method based on adaptive crossover point selection. At the selected crossover point, it swaps the node configurations of the two parent solutions and recalculates the total execution cost and total execution time of the solution. The formula for calculating the crossover point selection probability is as follows:

[0050]

[0051] In the formula, Indicates the selection of a location As the probability of choosing the intersection point For the intersection location index, For adaptive parameters, The length of the solution;

[0052] Topology repair mutation strategies include the selection of mutated nodes, the determination of nodes, and the repair process, targeting the decoded sequence. In the process of selecting mutation nodes, from a uniform distribution Randomly select a mutated node , This represents the total number of nodes in the decoded sequence; during the node determination process, it starts from the mutated nodes. A new edge is randomly selected from the adjacency list of the predecessor node to replace it. The edge replacement must satisfy the topological continuity constraint, that is, there exists an edge from the predecessor node to the target node of the new edge, and the target node of the new edge is not equal to the mutated node. The target node of the new edge represents the node that the new edge points to; during the repair process, if the new edge causes a mutated node in the decoded sequence... The following part is related to the target node. If connectivity breaks or topological continuity constraints are violated, a repair mechanism is triggered to check for a connection from the disconnected node to the target node. If a path does not exist, identify and remove nodes that do not meet the resource constraints. Starting from the disconnected node, select edges that meet the resource constraints from the disconnected node's adjacency list to construct a path to the next node, and so on, until the target node is reached. .

[0053] Furthermore, in step 7, the specific process of the heuristic search-guided uninjection strategy is as follows: every... The offspring population is non-dominated and sorted. From the first non-dominated front, solutions are selected in descending order of crowding distance, and the solution with the largest crowding distance is chosen as the representative solution. The injection interval is set according to a preset heuristic search; the total execution cost or total execution time of representative solutions is extracted as a constraint; with the goal of minimizing execution cost, heuristic search is used to obtain the solution with the minimum execution time under the constraint of the total execution cost of representative solutions; with the goal of minimizing execution time, heuristic search is used to search for the solution with the minimum execution cost under the constraint of the total execution time of representative solutions; the generated single-objective optimal solution is injected into the offspring population to replace solutions with higher ranks or smaller crowding distances in the non-dominated sorting, or to supplement regions with smaller crowding distances in the offspring population, thus updating the offspring population.

[0054] Furthermore, in step 8, the crowding distance selection method is as follows: the temporary population is sorted by non-dominated order, the temporary population is divided into multiple non-dominated fronts, the crowding distance of the solution is calculated for each non-dominated front, the solutions are selected in order of increasing level of non-dominated fronts, the crowding distance of each non-dominated front is sorted in order of decreasing level, and the solutions with larger crowding distances are selected first, until the number of selected solutions reaches the preset population size, and a new population is generated.

[0055] Furthermore, in step 9, when updating the global Pareto optimal solution set, the non-dominated front after the current iteration's non-dominated sorting is merged with the historical optimal solution set and non-dominated sorting is performed. The solutions in the first non-dominated front after non-dominated sorting are extracted as the current global Pareto optimal solution set. At the same time, the solution with the minimum execution cost and the solution with the minimum execution time in the current global Pareto optimal solution set are recorded.

[0056] This invention provides a serverless MapReduce multi-objective scheduling optimization method. Addressing the issues raised in the background art regarding the multi-objective conflict between execution cost and execution time of MapReduce jobs in serverless environments, the slow convergence of traditional multi-objective evolutionary algorithms under complex constraints, the tendency to generate solutions that do not meet resource constraints, and the uneven distribution of the global Pareto optimal solution set, this invention constructs a non-dominated sorting genetic algorithm framework that integrates heuristic search. Through a solution injection strategy guided by heuristic search, single-objective optimal solutions are injected into the current population, enhancing the exploration capability of non-dominated region boundaries, improving the uniformity of the global Pareto optimal solution set, and overcoming the drawback of traditional non-dominated sorting genetic algorithms easily getting trapped in local optima in complex graph-constrained problems. In summary, this invention has the positive effects of balancing the execution cost and execution time of MapReduce jobs, improving the quality of the global Pareto optimal solution set of multi-objective scheduling schemes, and enhancing the convergence speed and robustness of the algorithm. Attached Figure Description

[0057] Figure 1 This is a schematic diagram illustrating the implementation process of the present invention;

[0058] Figure 2 This is a schematic diagram of the directed acyclic graph encoding structure of the present invention;

[0059] Figure 3 This is a schematic diagram of the topology repair mutation strategy in this invention;

[0060] Figure 4 This is a comparison chart of the supervolume of the present invention and four existing comparison algorithms under different task scales;

[0061] Figure 5 This is a comparison chart of the reverse generation distance between the present invention and four existing comparison algorithms under different task scales;

[0062] Figure 6 This is a comparison chart of the hypervolume of the present invention and four existing comparison algorithms under different task sizes;

[0063] Figure 7 This is a comparison chart of the reverse generation distance between the present invention and four existing comparison algorithms under different task sizes. Detailed Implementation

[0064] like Figure 1 As shown, the specific implementation process of the serverless MapReduce multi-objective scheduling optimization method provided by the present invention is as follows.

[0065] Step 1: Initialize the heuristically search-guided non-dominated sorting genetic algorithm's running parameters, including population size, crossover rate, mutation rate, and heuristically search-guided injection interval. Calibration experiments are conducted on key parameters to adapt to serverless MapReduce multi-objective scheduling scenarios. The parameter calibration experiments use two multi-objective optimization standard indicators for evaluation: hypervolume and reverse generation distance. A higher hypervolume indicates better solution set quality, while a lower reverse generation distance indicates better solution set convergence. Since hypervolume and reverse generation distance are two conflicting objectives, pursuing a higher hypervolume may lead to an increase in the reverse generation distance, and vice versa. Therefore, parameter selection needs to balance these two objectives. Calibration experiments are conducted on the three key parameters, testing hypervolume and reverse generation distance under different values ​​of population size, crossover rate, and mutation rate. The experimental results are shown in Table 1.

[0066] Table 1 Summary of calibration experimental results for three key parameters

[0067]

[0068] Based on the data analysis in Table 1, since overvolume and reverse generation distance are two conflicting objectives, parameter selection requires a trade-off between the two objectives. For the population size parameter, when the population size is 20, the overvolume is highest (0.98479) but the reverse generation distance is relatively large (0.06813). When the population size is 50, the overvolume is 0.98476 and the reverse generation distance is 0.06787, achieving a better balance between overvolume and reverse generation distance. For the crossover rate parameter, when the crossover rate is 0.9, the overvolume is highest (0.98478) but the reverse generation distance is 0.06787. When the crossover rate is 0.8, the overvolume is 0.98476 and the reverse generation distance is 0.06791, achieving a better balance between overvolume and reverse generation distance. Regarding the mutation rate parameter, when the mutation rate is 0.5, the oversize is the highest (0.98479) but the reverse generation distance is 0.06785. When the mutation rate is 0.1, the oversize is lower (0.98472) and the reverse generation distance is larger (0.06795). When the mutation rate is 0.3, the oversize is 0.98477 and the reverse generation distance is 0.06793. Considering the balance of the two indicators and the robustness of the algorithm, the final parameter configuration is determined as follows: population size of 50, crossover rate of 0.8, and mutation rate of 0.15. For non-critical parameters, the injection interval guided by heuristic search is set to 10 (generations). This parameter controls the execution frequency of the heuristic search-guided de-injection strategy. Setting it to 10 (generations) means that the heuristic search-guided de-injection strategy is executed once every 10 generations. This parameter value is based on the common settings of periodic strategies in multi-objective evolutionary algorithms. Combined with the characteristics of serverless MapReduce multi-objective scheduling scenarios, setting it to 10 (generations) can avoid over-reliance on heuristic search while maintaining the algorithm's convergence speed, ensuring that the algorithm achieves a balance between exploration and utilization.

[0069] Step 2: Construct a directed acyclic graph (DAG) for resource allocation in the serverless MapReduce job. Specifically, the DAG model is constructed by modeling each solution as a source node in the DAG. To the target node The path, Indicates the source node, The target node is represented by [target node], and the other nodes in the directed acyclic graph are [other nodes]. It means that, among them, This represents the stage index of the resource decision-making phase. This represents the node index within the resource decision-making phase. Indicates the first Phase 1 1 node The values ​​are 1, 2, 3, 4, and 5. A value of 1 corresponds to the Mapper memory configuration phase; A value of 2 corresponds to the number of Mappers in the current stage. When the value is 3, it corresponds to the stage of the number of input objects for the Reducer; When the value is 4, it corresponds to the coordinator memory configuration stage; When the value is 5, it corresponds to the Reducer memory configuration stage; decoding sequence. = . These represent the nodes corresponding to each resource decision-making stage of the path, where... This indicates the node corresponding to the path during the Mapper memory configuration phase. This indicates the node corresponding to the path in the Mapper count stage. This indicates the node corresponding to the number of input objects in the Reducer's path. This indicates the node corresponding to the path during the coordinator's memory configuration phase. This indicates the node corresponding to the path in the Reducer memory configuration phase. , , , , These represent the sequence numbers of nodes within the corresponding stages, with each path corresponding to only one node in each resource decision stage. For example, in... Figure 2 In the illustrated embodiment, the node set during the Mapper memory configuration phase includes nodes , , , This indicates that the path has four available Mapper memory configuration options during the Mapper memory configuration phase. The value range is 1 to 4; the node set in the Mapper quantity stage contains nodes. , , This indicates that there are 3 available Mapper count options for the path during the Mapper count phase. The value range is 1 to 3; the node set of the input object number stage of the Reducer contains nodes. , , , This indicates that the path has four available options for the number of input objects for the Reducer during the Reducer input object count stage. The value range is 1 to 4; the node set in the coordinator memory configuration phase contains nodes , , , This indicates that there are four available coordinator memory configuration options for the path during the coordinator memory configuration phase. The value range is 1 to 4; the node set in the Reducer memory configuration phase contains nodes. , , , This indicates that there are four available Reducer memory configuration options during the Reducer memory configuration phase. The value range is from 1 to 4. In the directed acyclic graph, each edge connects two nodes in adjacent stages, representing the transition from resource allocation selection in one resource decision stage to resource allocation selection in another. Each edge stores the execution time and execution cost of the corresponding resource allocation. The validity of the solution is judged based on the fact that all adjacent nodes are connected by edges in the directed acyclic graph, and the cumulative execution time and cumulative execution cost do not exceed a preset threshold. Where, the cumulative execution cost represents the time and cost from the source node during the path search process. The cumulative execution time is the sum of the execution costs of all edges on the path to the current node, representing the execution time from the source node during the path search process. The sum of execution times for all edges on the path to the current node. This occurs when the path search is complete, i.e., the path originates from the source node. Reach the target node The cumulative execution cost equals the total execution cost. The cumulative execution time equals the total execution time.

[0070] Step 3: Generate the initial population using a hybrid initialization strategy. This strategy includes a heuristic search-guided process and a constraint-aware random generation process. In the heuristic search-guided process, heuristic search is used to generate solutions with minimum execution cost and minimum execution time. In the constraint-aware random generation process, different solutions are constructed using depth-first search and dynamic pruning methods.

[0071] Specifically, when using heuristic search to generate the solution with the minimum execution cost, an adaptive threshold method is adopted. The process involves starting from the source node... Initially, maintain an open list and a closed list. The open list stores nodes to be expanded, and the closed list stores expanded nodes. Each time, select the node with the smallest evaluation function value from the open list for expansion. The evaluation function is defined as follows:

[0072]

[0073] in Represents a node The evaluation function value, This represents a node in a directed acyclic graph. Indicates from the source node To the node The actual implementation cost Indicates from node To the target node The estimated execution cost, The relaxation factor is dynamically adjusted when the expanded node is the target node. When the backtracking path yields the solution with the minimum execution cost;

[0074] When using heuristic search to generate a solution with the minimum execution time, a reverse search approach is employed. The process involves starting from the target node... To the source node Perform a reverse search, maintaining an open list and a closed list. Each time, select the node with the shortest cumulative execution time from the open list for expansion. When the expanded node is the source node... When the time is right, the backtracking path yields the solution with the minimum execution time;

[0075] The specific process of depth-first search is as follows: starting from the source node... Begin by initializing the current node as the source node. The initial path is empty, and the source node is set. Add a path when the current node is not equal to the target node. When the current node is empty, obtain the adjacency list of the current node. The adjacency list stores all edges originating from the current node. If the adjacency list is empty, terminate the generation of the current path. Randomly and evenly select an edge from the adjacency list, update the target node connected to the selected edge to the current node, and add the execution cost and execution time of the current node and edge to the path.

[0076] The specific process of the dynamic pruning method is as follows: after each node expansion, calculate the number of nodes from the source node. The cumulative execution cost and cumulative execution time up to the current node, when the cumulative execution cost exceeds the budget limit. Or the cumulative execution time exceeds the deadline. If the current path cannot be expanded, backtrack to the last node in the path that satisfies the resource constraints. Remove the tried edges from the adjacency list of the last node that satisfies the resource constraints, and randomly select an edge that satisfies the resource constraints again for expansion. If no edge that satisfies the resource constraints can be found after backtracking, abandon the current path and start again from the source node. Start generating a new path.

[0077] Specifically, resource constraints include function concurrency limits, temporary storage limits, quality of service constraints, and selection variable constraints. The mathematical expression for function concurrency limits is:

[0078]

[0079] In the formula, This indicates the number of serverless functions used in the Map phase. This indicates the number of steps in the Reduce phase. Indicates the Reduce phase. The number of Reducers in each step This indicates the maximum concurrent resource limit allowed by the serverless platform;

[0080] The mathematical expression for the temporary storage limit is:

[0081]

[0082] In the formula, Indicates the index of the Map task. Indicates the first The size of intermediate results for each Map task, in MB. This indicates platform limitations;

[0083] Service quality constraints include task completion time constraints and execution cost constraints. The mathematical expression for the task completion time constraint is:

[0084]

[0085] In the formula, Indicates the total execution time. Indicates the deadline;

[0086] The mathematical expression for the execution cost constraint is:

[0087]

[0088] In the formula, Indicates the total execution cost. The mathematical expression representing budget constraints and selection variable constraints is:

[0089]

[0090]

[0091] in Indicates the number of optional configurations. Indicates selecting the first Indicator variables for various types of serverless functions Indicates selecting the first This memory type is used for the indicator variable of the Reducer. and Use binary variables to ensure the uniqueness of the schedule.

[0092] Step 4: Evaluate the fitness of each solution in the current population. The mathematical expression for the objective function that minimizes the task completion time is:

[0093]

[0094] in Let represent the objective function that minimizes the task completion time. Indicates the total execution time. Indicates the execution time of the Map phase. Indicates the execution time of the coordinator phase. Indicates the execution time of the Reduce phase;

[0095] The mathematical expression for the objective function that minimizes execution cost is:

[0096]

[0097] in This represents the objective function that minimizes the execution cost. Indicates the total execution cost. Indicates the cost of serverless function calls. Indicates the execution cost of a serverless function. Indicates the request cost of the object storage system. The object storage system storage cost represents intermediate data. The serverless function call cost refers to the fixed cost incurred when calling a serverless function. The serverless function execution cost refers to the cost calculated based on the execution time and memory configuration during the execution of a serverless function. The object storage system refers to a distributed storage system used to store intermediate data. The object storage system request cost refers to the request cost incurred when accessing the object storage system. The object storage system storage cost refers to the storage cost of intermediate data in the object storage system.

[0098] Step 5: Perform non-dominated sorting on the current population, initialize the dominance count based on dominance relationships, iteratively construct non-dominated fronts, and calculate the crowding distance of solutions in each non-dominated front. Specifically, the non-dominated sorting method is as follows: initialize the list of dominant solutions and the dominance count for each solution, compare any combination of two solutions in the current population, and if the solution... It is superior to or equal to the solution in both the objective function of minimizing task completion time and the objective function of minimizing execution cost. It outperforms at least one of the objective functions: minimizing task completion time and minimizing execution cost. ,but Dominate ,Will join in The list of dominant solutions will Increment the dominance count by 1, and add the solution with a dominance count of 0 to the first non-dominated frontier. A dominance count of 0 indicates that the solution is not dominated by any other solution. Iterate through the next non-dominated frontier. The first non-dominated frontier has the lowest level, and the level of the next non-dominated frontier increases. Traverse all solutions dominated by each solution in the current non-dominated frontier, and decrement the dominance count of each traversed solution by 1. When the dominance count of a traversed solution drops to 0, the traversed solution is added to the next non-dominated frontier.

[0099] The method for calculating the congestion distance is to sort the solutions within each non-dominated frontier in ascending order according to the objective function value that minimizes the task completion time, thus obtaining a sequence. The sequence is obtained by sorting the values ​​of the objective function that minimize execution cost in ascending order. Will The crowding distance of the solution that minimizes or maximizes the objective function value is set to infinity. The crowding distance of the solution that minimizes or maximizes the objective function value is set to infinity; for any other solution in this non-dominated front... Its crowding distance is defined as: in Winning The normalized distance between two adjacent solutions minimizes the objective function of task completion time, and in Winning The normalized distance between two adjacent solutions is calculated to minimize the execution cost objective function; the sum of the two is... Crowded distances.

[0100] Step 6: Execute tournament selection, adaptive crossover strategy, and topology repair mutation strategy sequentially to generate offspring population. Specifically, tournament selection randomly selects a preset number of solutions from the current population for comparison, and selects solutions with lower rank of non-dominated front and larger crowding distance as parent solutions;

[0101] The adaptive crossover strategy employs a crossover method based on adaptive crossover point selection. At the selected crossover point, it swaps the node configurations of the two parent solutions and recalculates the total execution cost and total execution time of the solution. The formula for calculating the crossover point selection probability is as follows:

[0102]

[0103] In the formula, Indicates the selection of a location As the probability of choosing the intersection point For the intersection location index, For adaptive parameters, The length of the solution;

[0104] Topology repair mutation strategies include the following processes, such as Figure 3 As shown, the source node and target node exist Figure 3 The nodes correspond to the source node and the target node, respectively. , , , , , , , , , , The nodes represent the resource decision stages along the path, with each path corresponding to only one node at each resource decision stage. The topology repair mutation strategy includes the selection of mutated nodes, the determination of nodes, and the repair process, given a solution encoding sequence. = In the process of selecting mutation nodes, from a uniform distribution Randomly select a mutated node , This represents the total number of nodes in the decoded sequence.

[0105] Specifically, in the process of selecting mutation nodes, such as Figure 3 As shown, the initial path is Among the nodes This represents the node corresponding to the initial path during the Mapper memory configuration phase. This represents the node corresponding to the initial path during the Mapper count phase. This represents the node corresponding to the initial path in the stage of the Reducer's input object count. Selected as a mutated node Figure 3 The dashed arrows in the path start from the nodes. Pointing to the node below The annotation "Select Mutant Nodes" indicates the process of selecting mutated nodes, and the nodes below... This refers to the selected mutated node.

[0106] In the process of determining nodes, such as Figure 3 As shown, from the mutated node A new edge is randomly selected from the adjacency list of the predecessor node to replace it. Figure 3 In the middle, node The node corresponding to the mutated node The predecessor node, Figure 3 The label "randomly select a new edge" indicates that an edge is randomly selected from multiple candidate edges. Figure 3 From the mutated node predecessor node There are three candidate edges pointing to the node. ,node ,node mutated nodes It will be replaced with any one of these three nodes, where the mutated node is... The selected path is indicated by a bold arrow. Edge replacement must satisfy the topological continuity constraint, that is, there exists an edge from the predecessor node to the target node of the new edge, and the target node of the new edge is not equal to the mutated node. The target node of the new edge represents the node that the new edge points to. Mutated node. It can mutate into a node. , , Any one of them, the selected path will mutate the node. Mutate into a node Each candidate mutated node has a corresponding subsequent path segment, which points to the node. ,node ,node The selected path starts from the node. Pointing to node , Figure 3 From the node To the node The path segment marked "connectivity check" indicates that a connectivity check is performed on that path segment.

[0107] During the repair process, such as Figure 3 As shown, the generated path undergoes constraint checks and repairs. From the node The path from the starting point to the destination node The path segment is marked "connectivity break," indicating that the new edge causes a mutated node in the decoded sequence. The following part is related to the target node. The connectivity break, that is, from the node To the target node The path is incomplete or does not exist. From node The path from the starting point to the destination node The path segment marked "Does not satisfy topological continuity constraint" indicates that the edge replacement violated the topological continuity constraint, meaning the newly selected edge does not meet the topological continuity requirement. From node The path from the starting point to the destination node The path segment marked "Reselect Edges to Rebuild Path" indicates that when a connectivity break or violation of topological continuity constraints is detected, a repair mechanism is triggered to check for a path from the disconnected node to the target node. If a path does not exist, identify and remove nodes that do not meet the resource constraints. Starting from the disconnected node, select edges that meet the resource constraints from the disconnected node's adjacency list to construct a path to the next node, and so on, until the target node is reached. .like Figure 3 As shown, all paths eventually converge, starting from node The path from the starting point to the target node From node The starting path also reaches the target node. This indicates that the repaired path may reach the target node through different resource decision-making stage nodes. .

[0108] Step 7: Periodically execute the heuristic search-guided solution injection strategy to inject the single-objective optimal solution into the offspring population. Specifically, every [percentage missing]... The offspring population is non-dominated and sorted. From the first non-dominated front, solutions are selected in descending order of crowding distance, and the solution with the largest crowding distance is chosen as the representative solution. The injection interval is set according to a preset heuristic search; the total execution cost or total execution time of representative solutions is extracted as a constraint; with the goal of minimizing execution cost, heuristic search is used to obtain the solution with the minimum execution time under the constraint of the total execution cost of representative solutions; with the goal of minimizing execution time, heuristic search is used to search for the solution with the minimum execution cost under the constraint of the total execution time of representative solutions; the generated single-objective optimal solution is injected into the offspring population to replace solutions with higher ranks or smaller crowding distances in the non-dominated sorting, or to supplement regions with smaller crowding distances in the offspring population, thus updating the offspring population.

[0109] Step 8: Merge the parent and offspring populations, and perform crowding distance selection on the merged temporary population to generate a new population. Specifically, the crowding distance selection method is as follows: perform non-dominated sorting on the temporary population, divide the temporary population into multiple non-dominated fronts, calculate the crowding distance of solutions for each non-dominated front, select solutions in ascending order of non-dominated front level, sort each non-dominated front by crowding distance from largest to smallest, and prioritize the selection of solutions with larger crowding distances, until the number of selected solutions reaches the preset population size, and generate a new population.

[0110] Step 9: Update the global Pareto optimal solution set. Specifically, when updating the global Pareto optimal solution set, merge the non-dominated front after the current iteration's non-dominated sorting with the historical optimal solution set and perform non-dominated sorting. Extract the solutions in the first non-dominated front after non-dominated sorting as the current global Pareto optimal solution set, and record the solution with the minimum execution cost and the solution with the minimum execution time in the current global Pareto optimal solution set.

[0111] Step 10: Determine if the time limit has been reached. If it has, output the current global Pareto optimal solution set and the total execution cost and total execution time corresponding to the current global Pareto optimal solution set. Otherwise, return to step 4 to continue iterating.

[0112] The technical effectiveness is verified through specific embodiments below. The experimental environment simulates the characteristics of a serverless platform, constructing 100 MapReduce job scheduling test instances, including 10 scheduling sizes (S1 to S10) and 10 task sizes (T1 to T10). Each combination of size and task size constitutes a test instance. The node processing rate is set according to the serverless platform. The experiment is repeated 15 times to reduce errors. The experimental platform is developed using Java SE 8, running on a Windows 10 (64-bit) operating system, with hardware configuration of an Intel® Core™ i7-12700 2.10 GHz CPU and 32 GB RAM. The evaluation metrics used are two multi-objective optimization standard indicators: hypervolume (HV) and inverse generation distance (IGD). A higher hypervolume indicates better solution set quality, while a lower inverse generation distance indicates better solution set convergence.

[0113] The heuristic search-guided non-dominated sorting genetic algorithm (ANSGA-II) of this invention was subjected to ablation experiments relative to the heuristic search-guided uninjection strategy, topology repair mutation strategy, and hybrid initialization strategy. Ablation variant 1 represents the algorithm variant of this invention that removes the heuristic search-guided uninjection strategy, ablation variant 2 represents the algorithm variant of this invention that removes the topology repair mutation strategy, and ablation variant 3 represents the algorithm variant of this invention that removes the hybrid initialization strategy. The hypervolume comparison of the ablation experiments is shown in Table 2.

[0114] Table 2 Comparison of overvolume ablation experiments

[0115]

[0116] Table 3 shows the comparison of reverse generation distances in ablation experiments:

[0117] Table 3 Comparison of reverse generational distances in ablation experiments

[0118]

[0119] The values ​​in Table 2 represent the average hypervolume of the ANSGA-II algorithm and three ablation variants after 15 repetitions at 10 different job scales. The values ​​in Table 3 represent the average reverse generation distance of the ANSGA-II algorithm and three ablation variants after 15 repetitions at 10 different job scales. Based on the data analysis in Tables 2 and 3, the ANSGA-II algorithm of this invention achieved optimal or near-optimal performance across all job scales. According to the data in Table 2, the hypervolume of the ANSGA-II algorithm ranges from 0.95012 to 0.99967, with an optimal value of 0.99967. According to the data in Table 3, the reverse generation distance of the ANSGA-II algorithm ranges from 0.02552 to 0.12962. All three ablation variants exhibited significant performance degradation. For example, in the S1 scale, according to Table 2, the hypervolume of ablation variant 1 was 0.93527, significantly lower than ANSGA-II's 0.95012; according to Table 3, the reverse generation distance of ablation variant 1 was 0.11059, significantly higher than ANSGA-II's 0.10385. In the S8 scale, according to Table 2, the hypervolume of ablation variant 2 was 0.93244, significantly lower than ANSGA-II's 0.97643; according to Table 3, the reverse generation distance of ablation variant 2 was 0.10992, significantly higher than ANSGA-II's 0.04798. In the S9 scale, according to Table 3, the reverse generation distance of ablation variant 2 reached 0.15708, significantly higher than ANSGA-II's 0.08980. These results demonstrate that removing any of the three strategies—heuristic search-guided solution injection, topology repair mutation, and hybrid initialization—leads to a significant decrease in algorithm performance. All three strategies play crucial roles in improving algorithm performance. Specifically, the heuristic search-guided solution injection and hybrid initialization strategies are key to enhancing algorithm stability, while the topology repair mutation strategy is essential for maintaining the feasibility and convergence of solutions. This invention combines the heuristic search-guided solution injection, topology repair mutation, and hybrid initialization strategies to propose a heuristic search-guided non-dominated sorting genetic algorithm (ANSGA-II), which solves the complex problem of solving serverless MapReduce multi-objective scheduling within a finite time limit.

[0120] By comparing the heuristic search-guided non-dominated sorting genetic algorithm (ANSGA-II) of this invention with four existing comparative algorithms, the advantages of this invention in solution set quality and convergence performance in serverless MapReduce multi-objective scheduling scenarios are verified. The four existing comparative algorithms are: decomposition-based multi-objective evolutionary algorithm (MOEA / D), multi-objective gray wolf optimization algorithm (MOGWO), multi-objective particle swarm optimization algorithm (MOPSO), and non-dominated sorting genetic algorithm (NSGA-II). Specific comparative experimental results are shown in Tables 4, 5, 6, and 7. Figure 4 , Figure 5 , Figure 6 , Figure 7 The comparison of the hypervolume of this invention with four comparison algorithms at different scales is shown in Table 4:

[0121] Table 4. Hypervolume comparison between the present invention and four comparative algorithms at different scales.

[0122]

[0123] As can be seen from Table 4, in all 10 job scale test instances, the hypervolume of the ANSGA-II of the present invention achieved the highest value relative to the other four comparison algorithms, and the value range was concentrated between 0.9972 and 0.9998. Compared with the four comparison algorithms, there were no obvious degradation points. This shows that the ANSGA-II of the present invention can continuously obtain a Pareto solution set with a more uniform distribution and a larger coverage under different job scales.

[0124] The inverse generation distance comparison between this invention and four comparison algorithms at different scales is shown in Table 5:

[0125] Table 5. Comparison of reverse generation distance between the present invention and four comparison algorithms at different scales.

[0126]

[0127] As shown in Table 5, in 10 job scale test instances, the ANSGA-II of the present invention consistently achieved the minimum reverse generation distance index compared to the other four comparison algorithms, and the value remained below 0.0033. In contrast, the reverse generation distance of the other four comparison algorithms increased significantly at some scales. This indicates that the ANSGA-II of the present invention is more likely to approximate the true Pareto front at different job scales and has better global convergence ability.

[0128] The supervolume comparison of this invention with four comparative algorithms under different task sizes is shown in Table 6:

[0129] Table 6. Hypervolume Comparison of the Invention and Four Comparison Algorithms under Different Task Sizes.

[0130]

[0131] As can be seen from the values ​​in Table 6, as the task size gradually increases from T1 to T10, the supervolume of the ANSGA-II of the present invention remains above 0.9974 under all task sizes and shows a slow upward trend overall. In contrast, the other four comparison algorithms show obvious deterioration when the task size is small, indicating that the present invention has better adaptability and scalability to different task sizes.

[0132] The inverse generation distance comparison between this invention and four comparison algorithms under different task sizes is shown in Table 7:

[0133] Table 7. Comparison of reverse generation distance between the present invention and four comparison algorithms under different task sizes.

[0134]

[0135] As shown in Table 7, in 10 test cases with different task sizes, the ANSGA-II of the present invention consistently outperforms the other four comparative algorithms in terms of the reverse generation distance index. Moreover, the reverse generation distance is less than 0.0031 for all task sizes from T1 to T10, while the other four comparative algorithms show a sudden increase in value for several task sizes. This indicates that the ANSGA-II of the present invention can still maintain stable convergence performance and solution set accuracy when the task size changes significantly.

[0136] Figure 4 The horizontal axis represents the job size, including 10 job size levels from S1 to S10, and the vertical axis represents the hypervolume; a higher value indicates better solution set quality. Figure 4 As can be seen, the hypervolume curve of this invention consistently ranks higher than the other four comparative algorithms and remains stable across different job scales, with the hypervolume consistently above 0.997, reaching a maximum of 0.9998. This demonstrates that ANSGA-II of this invention can obtain high-quality Pareto optimal solution sets under different job scales. In contrast, the hypervolume curves of the other four comparative algorithms, MOEA / D, MOGWO, MOPSO, and NSGA-II, fluctuate significantly and are considerably lower than those of ANSGA-II. Particularly in the job scale S8 test instance, MOPSO's hypervolume is only 0.9613, significantly lower than the 0.9998 of this invention, a difference of 3.85 percentage points, fully demonstrating the significant advantage of this invention in solution set quality.

[0137] Figure 5 The horizontal axis represents the job size, including 10 job size levels from S1 to S10, and the vertical axis represents the reverse generation distance. The lower the value, the better the convergence of the solution set. Figure 5 The graph shows a line graph illustrating the reverse generation distance between the present invention and four comparison algorithms as a function of job size. Figure 5 It can be seen that the reverse generation distance curve of the present invention is always below the other four comparison algorithms, and the curve is smooth and stable with small fluctuations. As the job scale increases from S1 to S10, the reverse generation distance curve of the present invention ANSGA-II remains basically horizontal with no obvious upward or downward trend. This shows that the present invention ANSGA-II can quickly converge to the Pareto optimal solution set under different job scales.

[0138] Figure 6 The horizontal axis represents the task size, including 10 task size levels from T1 to T10, and the vertical axis represents the hypervolume; a higher value indicates a better solution set quality. Figure 6 The graph shows the supervolume of this invention and four comparative algorithms as a function of task size. Figure 6 As can be seen, the hypervolume curve of this invention exhibits a stable upward trend. As the task size gradually increases from T1 to T10, the hypervolume curve of this invention steadily rises from 0.9974 to 0.9997, showing an overall monotonically increasing trend. It maintains a high level across different task sizes, with the hypervolume value consistently above 0.997, reaching a maximum of 0.9997. This indicates that this invention has good adaptability to different task sizes, can stably obtain high-quality Pareto optimal solution sets, and further improves the quality of the solution set as the task size increases. In contrast, the hypervolume curves of the other four comparison algorithms fluctuate significantly. The curves of MOEA / D and NSGA-II show a slow upward trend, but with a low starting point and obvious fluctuations. The curves of MOGWO and MOPSO fluctuate the most dramatically. Although the hypervolume of the comparison algorithms improves with increasing task size, it is still significantly lower than that of this invention, fully demonstrating the stability and superiority of this invention in terms of solution set quality.

[0139] Figure 7 The horizontal axis represents the task size, including 10 task size levels from T1 to T10, and the vertical axis represents the backward generation distance. The lower the value, the better the convergence of the solution set. Figure 7 The graph shows a line graph illustrating the reverse generation distance between the present invention and four comparison algorithms as a function of task size. Figure 7As can be seen, the backward generation distance curve of ANSGA-II of this invention is consistently lower than that of the other four comparative algorithms, and the curve is smooth and stable. As the task size gradually increases from T1 to T10, the backward generation distance curve of ANSGA-II of this invention fluctuates slightly between 0.0006 and 0.0031, maintaining a low level overall without a significant upward or downward trend. This indicates that this invention can quickly converge to the Pareto optimal solution set under different task sizes. In contrast, the backward generation distance curves of the other four comparative algorithms fluctuate more. The curves of MOEA / D and NSGA-II show a trend of first decreasing and then fluctuating. The curve of MOGWO fluctuates significantly within the range of T1 to T10. The curve of MOPSO fluctuates the most dramatically. As the task size increases, although the backward generation distance of the four comparative algorithms decreases, it is still significantly higher than that of ANSGA-II, and the fluctuation amplitude is large, fully demonstrating the significant advantages of ANSGA-II of this invention in terms of convergence speed and stability.

[0140] In summary, according to Table 4, Figure 4 and Table 5, Figure 5 Based on the data, under different operational scale test instances, the ANSGA-II of this invention outperforms the other four comparative algorithms in both hypervolume and reverse generation distance. In the operational scale S1 test instance, according to Table 4 and... Figure 4 According to the data, the hypervolume of ANSGA-II is 0.9993, which is higher than 0.9935 for the MOEA / D algorithm and 0.9811 for MOGWO; based on Table 5 and Figure 5 According to the data, the reverse generation distance of ANSGA-II is 0.0007, lower than MOEA / D's 0.0232 and MOGWO's 0.0269. In the S8 test instance, based on Table 4 and... Figure 4 The data shows that the hypervolume of ANSGA-II is 0.9998, MOEA / D is 0.9940, and MOPSO is 0.9613; according to Table 5 and Figure 5 According to the data, the reverse generation distance of ANSGA-II is 0.0013, MOEA / D is 0.0220, and MOPSO is 0.0505. Based on Table 6... Figure 6 and Table 7, Figure 7 Based on the data, under test instances of different task sizes, the ANSGA-II of this invention exhibits good robustness in both hypervolume and reverse generation distance metrics. According to Table 6 and... Figure 6 According to the data, the hypervolume of the ANSGA-II of this invention remained above 0.997 in all test cases, ranging from 0.9974 to 0.9997; based on Table 7 and Figure 7Based on the data, the reverse generation distance values ​​of ANSGA-II in this invention are distributed in the range of 0.0006 to 0.0031. Therefore, under different task sizes, the algorithm of this invention can converge to the Pareto optimal solution set. In contrast, the other four comparative algorithms have greater volatility. MOEA / D, according to Table 6 and Figure 6 The data shows that its supervolume fluctuates between 0.9898 and 0.9973; according to Table 7 and Figure 7 The data shows that the reverse generation distance value fluctuates between 0.0133 and 0.0371. Therefore, experimental data demonstrates that, compared with the other four comparison algorithms, the ANSGA-II of this invention improves the solution set quality and convergence of serverless MapReduce multi-objective scheduling, exhibits better stability and robustness, and is better adapted to the characteristics of serverless environments.

Claims

1. A serverless MapReduce multi-objective scheduling optimization method, characterized in that, Includes the following steps, Step 1: Initialize the running parameters of the heuristic search-guided non-dominated sorting genetic algorithm, including population size, crossover rate, mutation rate, and heuristic search-guided injection interval; Step 2: Construct a directed acyclic graph for resource allocation of serverless MapReduce jobs, and encode the resource allocation decision as a solution from the source node to the target node; Step 3: Generate the initial population using a hybrid initialization strategy. This strategy includes a heuristic search-guided process and a constraint-aware random generation process. In the heuristic search-guided process, heuristic search is used to generate solutions with minimum execution cost and minimum execution time. In the constraint-aware random generation process, different solutions are constructed using depth-first search and dynamic pruning methods. When using heuristic search to generate the solution with minimum execution cost, an adaptive threshold approach is employed. The process involves starting from the source node... Initially, maintain an open list and a closed list. The open list stores nodes to be expanded, and the closed list stores expanded nodes. Each time, select the node with the smallest evaluation function value from the open list for expansion. The evaluation function is defined as follows: ,in Represents a node The evaluation function value, This represents a node in a directed acyclic graph. Indicates from the source node To the node The actual implementation cost Indicates from node To the target node The estimated execution cost, The relaxation factor is dynamically adjusted when the expanded node is the target node. When the backtracking path yields the solution with the minimum execution cost; When using heuristic search to generate a solution with the minimum execution time, a reverse search approach is employed. The process involves starting from the target node... To the source node Perform a reverse search, maintaining an open list and a closed list. Each time, select the node with the shortest cumulative execution time from the open list for expansion. When the expanded node is the source node... When the time is right, the backtracking path yields the solution with the minimum execution time; The specific process of depth-first search is as follows: starting from the source node... Begin by initializing the current node as the source node. The initial path is empty, and the source node is set. Add a path when the current node is not equal to the target node. When the current node is empty, obtain the adjacency list of the current node. The adjacency list stores all edges originating from the current node. If the adjacency list is empty, terminate the generation of the current path. Randomly and evenly select an edge from the adjacency list, update the target node connected to the selected edge to the current node, and add the execution cost and execution time of the current node and edge to the path. The specific process of the dynamic pruning method is as follows: after each node expansion, calculate the number of nodes from the source node. The cumulative execution cost and cumulative execution time up to the current node, when the cumulative execution cost exceeds the budget limit. Or the cumulative execution time exceeds the deadline. If the current path cannot be expanded, backtrack to the last node in the path that satisfies the resource constraints. Remove the tried edges from the adjacency list of the last node that satisfies the resource constraints, and randomly select an edge that satisfies the resource constraints again for expansion. If no edge that satisfies the resource constraints can be found after backtracking, abandon the current path and start again from the source node. Start generating a new path; Step 4: Evaluate the fitness of each solution in the current population. Use the objective function of minimizing task completion time and the objective function of minimizing execution cost to evaluate each solution, and use the current population as the parent population. Step 5: Sort the current population by non-dominated ordering, initialize the dominance count based on the dominance relationship, iteratively construct the non-dominated front, and calculate the crowding distance of the solution in each non-dominated front; Step 6: The tournament selection, adaptive crossover strategy, and topology repair mutation strategy are executed sequentially to generate the offspring population. The topology repair mutation strategy includes a selection of mutation nodes, a node determination process, and a repair process, targeting the decoded sequence. = , These represent the nodes corresponding to the path at each resource decision stage. During the selection of mutated nodes, the distribution changes from uniform to uniform. Randomly select a mutated node , This represents the total number of nodes in the decoded sequence; during the node determination process, it starts from the mutated nodes. A new edge is randomly selected from the adjacency list of the predecessor node to replace it. The edge replacement must satisfy the topological continuity constraint, that is, there exists an edge from the predecessor node to the target node of the new edge, and the target node of the new edge is not equal to the mutated node. The target node of the new edge represents the node that the new edge points to; during the repair process, if the new edge causes a mutated node in the decoded sequence... The following part is related to the target node. If connectivity breaks or topological continuity constraints are violated, a repair mechanism is triggered to check for a connection from the disconnected node to the target node. If a path does not exist, identify and remove nodes that do not meet the resource constraints. Starting from the disconnected node, select edges that meet the resource constraints from the disconnected node's adjacency list to construct a path to the next node, and so on, until the target node is reached. ; Step 7: Periodically execute the heuristic search-guided solution injection strategy to inject the single-objective optimal solution into the offspring population, thus updating the offspring population; the specific process of the heuristic search-guided solution injection strategy is as follows: Every The offspring population is non-dominated and sorted. From the first non-dominated front, solutions are selected in descending order of crowding distance, and the solution with the largest crowding distance is chosen as the representative solution. The injection interval is set according to a preset heuristic search; the total execution cost or total execution time of representative solutions is extracted as a constraint; with the goal of minimizing execution cost, heuristic search is used to obtain the solution with the minimum execution time under the constraint of the total execution cost of representative solutions; with the goal of minimizing execution time, heuristic search is used to search for the solution with the minimum execution cost under the constraint of the total execution time of representative solutions; the generated single-objective optimal solution is injected into the offspring population to replace solutions with higher levels or smaller crowding distances in the non-dominated sorting, or to supplement regions with smaller crowding distances in the offspring population, and to update the offspring population. Step 8: Merge the parent and offspring populations, perform crowding distance selection on the merged temporary population, and generate a new population; Step 9: Update the global Pareto optimal solution set; Step 10: Determine if the time limit has been reached. If it has, output the current global Pareto optimal solution set and the total execution cost and total execution time corresponding to the current global Pareto optimal solution set. Otherwise, return to step 4 to continue iterating.

2. The serverless MapReduce multi-objective scheduling optimization method according to claim 1, characterized in that, In step 2, the method for constructing the directed acyclic graph model is to encode the resource allocation decision as a solution structure, and model each solution as a source node in the directed acyclic graph. To the target node The path, Indicates the source node, The target node is represented by [target node], and the other nodes in the directed acyclic graph are [other nodes]. It means that, among them, This represents the stage index of the resource decision-making phase. This represents the node index within the resource decision-making phase. Indicates the first Phase 1 1 node The values ​​are 1, 2, 3, 4, and 5. A value of 1 corresponds to the Mapper memory configuration phase; A value of 2 corresponds to the number of Mappers in the current stage. When the value is 3, it corresponds to the stage of the number of input objects for the Reducer; When the value is 4, it corresponds to the coordinator memory configuration stage; When the value is 5, it corresponds to the Reducer memory configuration stage; decoding sequence. = , These represent the nodes corresponding to each resource decision stage of the path, with each path corresponding to only one node in each stage. Each edge in the directed acyclic graph connects two nodes in adjacent stages, representing the transition from resource configuration selection in one stage to selection in another. Each edge stores the execution time and cost of the corresponding resource configuration. The validity of a solution is determined based on the fact that all adjacent nodes are connected by edges in the directed acyclic graph, and the cumulative execution time and cost do not exceed a preset threshold. The cumulative execution cost represents the time and cost from the source node during the path search process. The cumulative execution time is the sum of the execution costs of all edges on the path to the current node, representing the execution time from the source node during the path search process. The sum of execution times for all edges on the path to the current node. When the path search is complete, i.e., the path originates from the source node... Reach the target node At that time, the cumulative execution cost equals the total execution cost, and the cumulative execution time equals the total execution time.

3. The serverless MapReduce multi-objective scheduling optimization method according to claim 2, characterized in that, Resource constraints include function concurrency limits, temporary storage limits, quality of service constraints, and selection variable constraints. The mathematical expression for the function concurrency limit is: In the formula, This indicates the number of serverless functions used in the Map phase. This indicates the number of steps in the Reduce phase. Indicates the Reduce phase. The number of Reducers in each step This indicates the maximum concurrent resource limit allowed by the serverless platform; The mathematical expression for the temporary storage limit is: In the formula, Indicates the index of the Map task. Indicates the first The size of intermediate results for each Map task, in MB. This indicates platform limitations; Service quality constraints include task completion time constraints and execution cost constraints. The mathematical expression for the task completion time constraint is: In the formula, Indicates the total execution time. Indicates the deadline; The mathematical expression for the execution cost constraint is: In the formula, Indicates the total execution cost. The mathematical expression representing budget constraints and selection variable constraints is: , ,in Indicates the number of optional configurations. Indicates selecting the first Indicator variables for various types of serverless functions Indicates selecting the first This memory type is used for the indicator variable of the Reducer. and Use binary variables to ensure the uniqueness of the schedule.

4. The serverless MapReduce multi-objective scheduling optimization method according to claim 3, characterized in that, In step 4, the mathematical expression for the objective function that minimizes the task completion time is: ,in Let represent the objective function that minimizes the task completion time. Indicates the total execution time. Indicates the execution time of the Map phase. Indicates the execution time of the coordinator phase. Indicates the execution time of the Reduce phase; The mathematical expression for the objective function that minimizes execution cost is: ,in This represents the objective function that minimizes the execution cost. Indicates the total execution cost. Indicates the cost of serverless function calls. Indicates the execution cost of a serverless function. Indicates the request cost of the object storage system. The object storage system storage cost represents intermediate data. The serverless function call cost refers to the fixed cost incurred when calling a serverless function. The serverless function execution cost refers to the cost calculated based on the execution time and memory configuration during the execution of a serverless function. The object storage system refers to a distributed storage system used to store intermediate data. The object storage system request cost refers to the request cost incurred when accessing the object storage system. The object storage system storage cost refers to the storage cost of intermediate data in the object storage system.

5. The serverless MapReduce multi-objective scheduling optimization method according to claim 4, characterized in that, In step 5, the non-dominated sorting method involves initializing the dominant solution list and dominance count for each solution, comparing any two solutions in the current population, and if the solution... It is superior to or equal to the solution in both the objective function of minimizing task completion time and the objective function of minimizing execution cost. It outperforms at least one of the objective functions: minimizing task completion time and minimizing execution cost. ,but Dominate ,Will join in The list of dominant solutions will The dominance count is incremented by 1. Solutions with a dominance count of 0 are added to the first non-dominated frontier. A dominance count of 0 indicates that a solution is not dominated by any other solution. The next non-dominated frontier is calculated iteratively. All solutions dominated by each solution in the current non-dominated frontier are traversed. The dominance count of each traversed solution is decremented by 1. When the dominance count of a traversed solution drops to 0, the traversed solution is added to the next non-dominated frontier. The first non-dominated frontier has the lowest level, and the level of the next non-dominated frontier increases. The crowding distance is calculated as follows: for each non-dominated frontier, sort the solutions according to the values ​​of the objective function that minimizes task completion time and the objective function that minimizes execution cost, and obtain a sorted sequence. Set the crowding distance of the solution with the smallest or largest value on the objective function that minimizes task completion time in the non-dominated frontier to infinity, and set the crowding distance of the solution with the smallest or largest value on the objective function that minimizes execution cost in the non-dominated frontier to infinity. The crowding distance of other solutions in the non-dominated frontier is the sum of the normalized distances of the solutions at adjacent positions in the sorted sequence on the objective functions that minimize task completion time and minimize execution cost.

6. The serverless MapReduce multi-objective scheduling optimization method according to claim 5, characterized in that, In step 6, the tournament selects a preset number of solutions from the current population for comparison, and selects the solution with a lower rank and larger crowding distance on the non-dominant front as the parent solution. The adaptive crossover strategy employs a crossover method based on adaptive crossover point selection. At the selected crossover point, it swaps the node configurations of the two parent solutions and recalculates the total execution cost and total execution time of the solution. The formula for calculating the crossover point selection probability is as follows: In the formula, Indicates the selection of a location As the probability of choosing the intersection point For the intersection location index, For adaptive parameters, Let be the length of the solution.

7. The serverless MapReduce multi-objective scheduling optimization method according to claim 6, characterized in that, In step 8, the crowding distance selection method is as follows: the temporary population is sorted by non-dominated order, the temporary population is divided into multiple non-dominated fronts, the crowding distance of the solution is calculated for each non-dominated front, the solutions are selected in order of increasing level of non-dominated fronts, the crowding distance of each non-dominated front is sorted in order of decreasing level, and the solutions with larger crowding distances are selected first, until the number of selected solutions reaches the preset population size, and a new population is generated.

8. The serverless MapReduce multi-objective scheduling optimization method according to claim 7, characterized in that, In step 9, when updating the global Pareto optimal solution set, the non-dominated front after the current iteration's non-dominated sorting is merged with the historical optimal solution set and non-dominated sorting is performed. The solutions in the first non-dominated front after non-dominated sorting are extracted as the current global Pareto optimal solution set. At the same time, the solution with the minimum execution cost and the solution with the minimum execution time in the current global Pareto optimal solution set are recorded.

Images