Large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization

Abstract

The invention provides a large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization, a system model of a Hadoop cluster comprises a plurality of racks, a plurality of physical machines (PMs) are configured in each rack, and a plurality of virtual machines (VMs) run on each physical machine; the Hadoop cluster performs data block transmission and file copy copying through a switch, so that cross-rack data synchronization and load balancing are realized; the optimization objective of the system model of the Hadoop cluster is to minimize the weighted sum of energy consumption, resource waste and file access delay of the Hadoop cluster, a variable-length chromosome structure composed of virtual machine chromosomes and block chromosomes is used in the method adopted by the optimization objective, and the chromosome structure is used for expressing the positions where virtual machines and data block copies are placed. A chromosome structure uses crossover and mutation operation to enhance diversity of solution space exploration of resource configuration in a virtual machine placement process; according to the method, dynamic changes of resource requirements in the Hadoop cluster operation process are fully considered, and it can be ensured that the Hadoop cluster keeps good performance under the high-load condition.

Description

Technical Field

[0001] The invention relates to the technical field of cloud computing, and in particular to a large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization. Background Art

[0002] With the rapid development of cloud computing, virtualization technology has become the core support of cloud data centers. By accessing cloud computing services, users can access and utilize resources (such as CPU / GPU, memory, and storage) on demand. This on-demand service approach significantly improves the resource utilization of cloud data centers while also reducing their operating costs. Therefore, more and more users upload their computing tasks to the cloud to cope with the growing demand for data processing.

[0003] As a classic distributed framework, Hadoop has been widely used in cloud environments due to its excellent scalability and flexibility. As one of Hadoop's core technologies, the distributed file system (Hadoop Distributed File System, HDFS) divides files into multiple data blocks and stores them on multiple data nodes, thereby ensuring high data availability and fault tolerance. As another core technology of Hadoop, MapReduce schedules computing tasks to the nodes where the data blocks are located to maximize data locality and reduce data transmission latency. However, with the continuous growth of data volume and cluster size, how to efficiently optimize virtual machine placement to improve cluster resource utilization and task processing efficiency has become a core issue in Hadoop cluster performance optimization. Virtual machine placement strategy directly affects key performance indicators such as Hadoop cluster energy consumption, resource utilization, and file access latency. Therefore, how to efficiently manage resources in large-scale dynamic Hadoop clusters has become an open research hotspot.

[0004] Traditional virtual machine placement mostly adopts strategies based on genetic algorithms, which show good performance when facing small-scale clusters, but expose limitations when facing large-scale clusters and variable loads, and cannot adapt well to dynamically changing resource requirements. To solve this problem, some studies have proposed multi-objective optimization algorithms, such as the Multi-Objective Particle Swarm Optimization (MOPSO) algorithm and the Multi-Objective Simulated Annealing (MOSA) algorithm. Specifically, the MOPSO algorithm simulates group behavior to perform global search, which is easy to fall into local optimality in complex environments and is sensitive to hyperparameters, so the strategy optimization process is difficult to control. The MOSA algorithm simulates the energy changes in the physical annealing process to find the global optimal solution, which can avoid falling into the local optimality, but the convergence speed is slow, and it will cause excessive system overhead when dealing with multi-objective optimization problems. In addition, the above algorithms cannot effectively handle multi-dimensional heterogeneous resources in the virtual machine placement problem. In order to solve the above important challenges, the present invention proposes a novel Multi-objective Optimization method with Variable Length Double chromosome (MO-VLD) method for large-scale Hadoop cluster virtual machine placement. The method considers energy consumption, resource utilization and file access latency as multiple optimization objectives. In particular, the proposed system model fully considers the dynamic changes of resource requirements to ensure that the Hadoop cluster can still maintain good performance under high load conditions. Summary of the invention

[0005] The present invention proposes a large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization, which regards energy consumption, resource utilization and file access latency as multiple optimization objectives. The system model used in the method fully considers the dynamic changes of resource requirements to ensure that the Hadoop cluster can still maintain good performance under high load conditions.

[0006] The present invention adopts the following technical solutions.

[0007] A large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization is disclosed. The placement method is used to form a Hadoop cluster. The system model of the Hadoop cluster includes multiple racks, each rack is configured with multiple physical machines PMs (Physics Machines, PMs), and each physical machine runs multiple virtual machines VMs (Virtual Machines, VMs); the racks are interconnected through core switches, and the PMs inside the racks are interconnected through switches; the Hadoop cluster transmits data blocks and replicates file copies through the switches, thereby realizing data synchronization and load balancing across racks; the optimization goal of the system model of the Hadoop cluster is to minimize the weighted sum of energy consumption, resource waste and file access delay of the Hadoop cluster, and the method adopted by the optimization goal uses a variable-length chromosome structure composed of virtual machine chromosomes and block chromosomes, the chromosome structure is used to express the location of placing virtual machines and data block copies, and the chromosome structure uses crossover and mutation operations to enhance the diversity of solution space exploration of resource configuration in the Hadoop cluster virtual machine placement process.

[0008] In the system model of Hadoop cluster, there are M physical machines in the Hadoop cluster, denoted as PM i (i=1,2,...M), its resource configuration is recorded as a four-tuple Respectively represent the CPU, memory, disk I / O, and network bandwidth of the PM; each PM i Run different numbers of VMs that share the same PM i resources; each VM j The resource usage of (j=1,2,...N) is recorded as a four-tuple The resources allocated to the virtual machine are limited by the resource configuration of the physical machine, which is expressed as In the system model of the Hadoop cluster, the sum of the resources allocated to the virtual machine and the virtualization software overhead must not exceed the resource configuration of the physical machine to avoid excessive consumption of the resources of the physical machine, which is defined by the formula

[0009] Among them, ∑ pmi U pmi Indicates PM i The sum of resource consumption of all virtual machines on the network; Indicates PM i Resource consumption of virtualization software.

[0010] In the system model of Hadoop cluster, VM j The resource consumption of a cannot exceed its allocated resources, which is defined by the formula

[0011]

[0012] In the system model of Hadoop cluster, the storage resource model is as follows: Suppose the file is divided into multiple data blocks B k (k=1,2,...,K) so that its copies are stored on different virtual machines, that is, the file is represented as a connection of blocks, defined by the formula

[0013]

[0014] The file size fSize fu The size of the data block fbSize fu Determines the number of data blocks, which is defined as

[0015] In order to improve data availability and system fault tolerance, each data block is stored in multiple copies on different virtual machines. k ; File f u Each data block There are multiple copies, defined as

[0016]

[0017] Among them, fR fu Represents file f u The number of copies of .

[0018] PM i Multiple copies of the same data block cannot be stored simultaneously to avoid wasting storage resources. This constraint is expressed as

[0019]

[0020] In the system model of the Hadoop cluster, the energy consumption model is as follows:

[0021] Set PM i The energy consumption of PM is affected by CPU, network bandwidth and disk I / O utilization. i The total energy consumption is defined as

[0022]

[0023] in, and They represent the energy consumption of CPU, network bandwidth, and disk I / O respectively;

[0024] Specifically, the CPU energy consumption is defined as follows:

[0025]

[0026] Among them, α i and β iRespectively represent PM i Low CPU load and high CPU load level coefficient; Prh = P max -P idle Indicates the difference between the highest and lowest power of a physical machine; λ i Indicates PM i When the load exceeds the threshold, the CPU energy consumption model will be switched from the low-load CPU energy consumption model to the high-load CPU energy consumption model.

[0027] The network bandwidth energy consumption is defined as

[0028]

[0029] in, represents the network bandwidth utilization, γ i represents the influence coefficient of network bandwidth utilization on energy consumption; disk I / O energy consumption is defined as

[0030]

[0031] in, Indicates disk I / O utilization, δ i Indicates the impact coefficient of disk I / O utilization on energy consumption.

[0032] In the system model of Hadoop cluster, physical machines provide computing, storage and network resources for multiple virtual machines. During the operation of Hadoop cluster, resource waste caused by inefficient or unreasonable resource utilization is expressed as the resource waste related formula in the resource utilization model:

[0033]

[0034] in, Indicates PM i The resource utilization rate includes CPU, disk bandwidth and network bandwidth utilization; ∈ is a positive coefficient, which is used to avoid unreasonable extreme situations when the standard deviation is zero; Indicates PM i The standard deviation of the utilization of each resource is used to measure the imbalance of resources and is defined as

[0035] in,

[0036] Represents the mean value of each resource utilization, which is defined as

[0037]

[0038] Therefore, PM i The sum of the utilization of all resources is defined as

[0039]

[0040] Assume that in the system model of the Hadoop cluster, the transmission of file data between physical machines needs to pass through multiple switches and physical links. The system model uses switch delay and physical link delay to evaluate the file access delay between virtual machines. The file access delay model of the system model is specifically:

[0041] Assume the switch delay T switch is the processing time required for the data block to pass through the switch, and the file access delay is the physical machine PM where the virtual machine is located. dn Go to the physical machine PM where the target virtual machine is located vm The sum of all switch delays and physical link delays between them is defined as

[0042]

[0043] Where N represents the number of switches in the transmission path, represents the delay of the kth physical link; when there are I data blocks, the average transmission delay of all data blocks is L avg Expressed as

[0044]

[0045] The optimization goal of the Hadoop cluster system model is an NP-hard multi-objective optimization problem. When solving the multi-objective optimization problem, the formal definition of the optimization problem is:

[0046] min(w1Z1+w2Z2+w3Z3)st(1),(2),(6)(17)

[0047] in, w1, w2 and w3 represent the weights of the three optimization goals of energy consumption, resource waste and file access delay respectively; the chromosome C of the virtual machine vm is a two-dimensional array, defined as

[0048]

[0049] The first row of each column indicates the physical machine where the virtual machine is placed, and the second row of each column indicates the type of the virtual machine, denoted as VMType. j ∈{1,2,3,4,5}.

[0050] Block chromosome C block is a one-dimensional array, indicating the placement of the data block copy, which is defined as

[0051] C block = {VM1, VM2, ... , VMn} (19)

[0052] Among them, each element represents the virtual machine where the data block replica is placed;

[0053] In the initialization phase, C vm Place virtual machines on physical machines in a round-robin manner and randomly assign virtual machine types; C block By generating data block copies in a round-robin and random manner, the distribution of data block copies is ensured to meet the requirements of the Hadoop cluster for the number of copies. The initialization process ensures the diversity of the solution space, so that the MO-VLD method for solving multi-objective optimization problems can fully explore different strategies during the operation phase.

[0054] During the evolution of the chromosome structure, a two-stage crossover and mutation operation is used to generate new solutions and optimize the existing solution set; the two stages are:

[0055] The first stage: single-point crossover of the virtual machine chromosome; randomly select the cutting point p, and exchange the fragments of different parent virtual machine chromosomes to generate the offspring virtual machine chromosome; the process is expressed as

[0056] The second stage: single-point crossover of block chromosomes; randomly select the cutting point q, and exchange the fragments of different parent block chromosomes to generate daughter block chromosomes; the process is expressed as

[0057]

[0058] The mutation operation of the chromosome structure refers to increasing the diversity of solutions by randomly changing the genes in the chromosome;

[0059] In the chromosome structure, the mutation operations on the virtual machine and block chromosomes are defined as follows: For the virtual machine chromosome, in the virtual machine mutation stage, the strategy used to enhance the diversity and optimization ability of the population is as follows: First, the expansion and contraction mutations of the virtual machine adapt to the dynamic resource demand by adjusting the number of virtual machines; the number of virtual machines is first set to The probability of increasing or decreasing provides greater resource configuration flexibility for the solution; then, the virtual machine replacement mutation traverses each element position of the virtual machine chromosome and randomly modifies its value with a probability set to 0.5; on average, this mutation operation will cause half of the virtual machine allocation and template types to change; the method of traversing the virtual machine chromosome includes allocation and template;

[0060] For block chromosomes, a block replacement operation is introduced: by changing the location of data block replicas to optimize their storage location, thereby reducing data transmission latency. Specifically, when a data block replica is located in a certain virtual machine, the block replacement operation will reallocate the replica to other virtual machines with a probability set to 0.5.

[0061] When solving multi-objective optimization problems, the target space is divided into multiple regions, each of which corresponds to a reference point. The reference points are associated and normalized. The reference points are predefined target vectors, which represent potential solutions under different target weights. Specifically, in the evolution of each generation of population, the current solution set will be projected onto the pre-generated reference points and associated according to the distance between the two. Then, the nearest reference point will attract the solution to move in its direction, and then the solution associated with the sparse reference point will be preferentially selected. The selection operator based on the reference point is used to maintain the diversity of the population and ensure the uniform distribution of the solutions, so as to avoid the concentration of solutions in certain target areas and ignore the optimization of other dimensions.

[0062] When solving multi-objective optimization problems, in order to compare different optimization objectives on the same scale, a normalization operation is introduced, specifically: converting individual performance indicators in energy consumption, resource utilization, and file access latency to the same magnitude, thereby eliminating the magnitude differences between different optimization objectives. This process is defined as

[0063]

[0064] Among them, r i represents the weighted target value of the i-th solution; r 1,i 、r 2,i and r 3,i represent energy consumption, resource waste and file access delay respectively; n represents the total number of Pareto solutions.

[0065] The MO-VLD method achieves global optimization of multi-dimensional objectives by combining the reference point association mechanism with the normalization operation.

[0066] The MO-VLD method comprises the following steps:

[0067] Step 1: Construction of chromosomes. When initializing the entire population, generate a combination of decision variables for each individual (chromosome) in the cluster according to the established process; randomly determine the number of virtual machines vmNumber according to the system configuration to ensure that it is not less than the number of physical machines, and then use the double loop and round-robin allocation strategy to generate the data block allocation plan, and add a random shuffle process at the appropriate position to ensure the diversity of solutions. Specifically: through the generateRoundRobin2Dshuffle function, first use generateRoundRobin1D to generate a list of continuous values ​​in a round-robin manner in one dimension, and then divide the list into multiple sub-segments according to the specified block size (ie, the replication factor), randomly shuffle each sub-segment, and then shuffle all sub-segment lists twice, finally forming a two-dimensional structure allocation pattern. After that, the two-dimensional structure list is serialized into a one-dimensional list through the serialize2D function for subsequent processing and storage;

[0068] Step 2: Generate corresponding sequences for virtual machines vm and virtual machine types vmtype, and use the same round-robin strategy (generateRoundRobin1D) as in step 1 to allocate a virtual machine ID list based on the range of physical machines and a virtual machine type list based on the number of available types (here 4) to vmNumber virtual machines. Then, randomly shuffle the two lists to obtain a decentralized and diverse virtual machine mapping and type allocation scheme in the initial state.

[0069] Step 3: After completing the construction and processing of block, vm, and vmtype, integrate them into a chromosome dictionary, which represents the complete chromosome structure of the individual. Then, by calling the removeEmptyVms function, clean up the generated chromosome (for example, remove unused virtual machine entries) to ensure the validity of the chromosome structure. Finally, add the chromosome to the solution list of the population. At the same time, initialize the dominance relationship and crowding distance and other multi-objective optimization related information for each solution in the population. The corresponding algorithm is expressed in pseudo code as follows:

[0070]

[0071] The corresponding algorithm process is as follows:

[0072] First, the initial maximum time slot T, the maximum number of iterations X, and the population size N are initialized (line 1), and the file list is read from the dataset of the Google cluster (line 2) to simulate the load demand of a real cloud data center;

[0073] In each time slot, the system environment is first initialized, including the configuration information of the physical machine and the virtual machine. In the con_system method, the file size list and the file access rate list are first initialized. Then, according to the file access rate and the predefined resource usage rate, the resource load of each file block is calculated. Specifically, the CPU demand is obtained by multiplying the file access rate by the predefined CPU usage rate. The IO demand is multiplied by a random number based on the CPU demand to simulate the volatility of IO resource usage. The network demand is also multiplied by a random number to simulate the volatility of network resource usage. For each file block and its replica, the calculated CPU, IO and network resource requirements are added to the block_load_list (line 4). In each round of iteration, the population is generated and initialized using the rotation and randomization methods to ensure the diversity of the solution set. During the initialization process, the physical bound load calculation method is used. The system traverses each virtual machine and accumulates the resource requirements of the virtual machine to the corresponding physical machine according to the physical machine index assigned to it, thereby obtaining the total resource usage of each physical machine. To ensure the rationality of resource allocation and the stability of the system, the resource constraint check implemented by the system is as follows: If any resource ( If the usage of a virtual machine or physical machine exceeds its configured capacity (CPU, IO or network), the solution of the corresponding virtual machine placement method will be considered infeasible (line 7); then, a single-point crossover is performed on the virtual machine and block chromosomes, and different mutation operations and block replacement operations are performed to increase population diversity; the virtual machine growth and shrinking methods are used to adjust the number of virtual machines to explore different virtual machine configurations and seek better resource allocation solutions; the virtual machine growth method provides more resource allocation options by increasing the number of virtual machines, which may reduce the load of a single virtual machine; while the virtual machine shrinking method simplifies resource management by reducing the number of virtual machines, but it is necessary to reallocate the data blocks on the deleted virtual machine to other virtual machines to avoid resource overload; when performing the virtual machine shrinking operation, the system first releases the CPU, IO and network resources occupied by the deleted virtual machine; then, the system reallocates the data blocks on the deleted virtual machine to other virtual machines, and through this process affects the CPU, IO and network resource usage of the receiving virtual machine. The system needs to ensure that after the reallocation, the new virtual machine will not affect the overall performance due to resource overload and keep the resource usage within an acceptable range; after the mutation operation, the resource allocation of the solution changes;In order to ensure that the new solution still meets the resource constraints and can accurately evaluate its fitness, the system recalculates the CPU, IO and network resource usage of each virtual machine and physical machine based on the current solution. The method includes traversing all file blocks, accumulating the resource requirements of the virtual machine assigned to each block, and then accumulating the resource usage of the physical machine according to the physical machine assigned to the virtual machine (lines 8-9). Subsequently, the fitness of the offspring is evaluated and merged with the original population, and resource constraints are checked. The system traverses all physical machines to compare whether their CPU, IO and network resource usage exceeds the configured capacity, and then traverses all virtual machines to compare whether their CPU, IO and network resource usage exceeds the configured capacity, so that the system ensures that each mutated solution is reasonable in resource allocation and achieves the optimization goal in overall performance and resource utilization (lines 10-11);

[0074] In order to guide the population to approach the Pareto optimal frontier, pre-generated reference points are introduced, the current solution set is projected onto these reference points, the non-dominated sorting and crowding distance are calculated, and then the next generation population is selected based on the correlation between the two (lines 12-13). To ensure the convergence and global search capabilities of the method, the fitness of the initial population is saved in the initialization phase, and the population quality is improved through multiple iterations. Then, the optimal solution is selected from the merged population as the next generation population, and the current Pareto frontier is saved (lines 14-15). Finally, normalization is used to ensure the balance between different optimization objectives and select the best virtual machine placement strategy (lines 17-18).

[0075] The method for using the virtual machine placement method comprises the following steps:

[0076] Step S1: first deploy a virtual machine resource management module in a large-scale Hadoop cluster environment, and configure initial resource information of physical machines and virtual machines, including CPU, memory, disk I / O, and network bandwidth;

[0077] Step S2: The algorithm used by the MO-VLD method monitors the resource demand changes and task load in the cluster in real time, and loads the corresponding file list or data task to ensure accurate grasp of the actual situation of the cluster, laying the foundation for subsequent optimization;

[0078] Step S3: In resource management, the virtual machine placement method dynamically adjusts the allocation of virtual machines and the placement of data blocks according to the current task load and cluster resource usage. Through intelligent adjustment, the cluster can maximize resource utilization efficiency and reduce energy consumption under different loads;

[0079] Step S4: During the optimization process, the virtual machine placement method continuously analyzes the operation effect of the cluster, and performs strategy feedback and adaptive optimization based on key indicators such as energy consumption, resource utilization, and data access latency to ensure that the cluster system is always in the best operating state.

[0080] The present invention proposes a large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization, using a new multi-objective optimization method with variable length double chromosome (MO-VLD) method for large-scale Hadoop cluster virtual machine placement. The method regards energy consumption, resource utilization and file access delay as multiple optimization objectives. In particular, the system model adopted by the proposed method fully considers the dynamic changes of resource requirements, which can ensure that the Hadoop cluster can still maintain good performance under high load conditions.

[0081] The present invention realizes efficient allocation of virtual machine resources in large-scale Hadoop clusters by combining variable-length double chromosomes and multi-objective optimization algorithms, significantly improves resource utilization and reduces overall energy consumption, has strong dynamic adaptability, and can automatically adjust allocation strategies according to changes in actual data load and resource requirements to ensure stable and efficient operation of the cluster. The invention adopts a non-dominated sorting genetic algorithm (NSGA-III) to achieve an efficient balance between resource utilization, energy consumption and access latency, has excellent global optimization capabilities, and has good flexibility and scalability. It is suitable for Hadoop clusters of various sizes, from small clusters to large cloud data centers, and can provide stable virtual machine placement optimization solutions. The main application scenarios include large-scale data processing environments, virtualized resource management of cloud computing platforms, and complex computing scenarios that require dynamic resource scheduling, such as edge computing and cloud-edge collaborative computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0083] Attached Figure 1 It is a schematic diagram of the architecture of the present invention;

[0084] Attached Figure 2 It is a schematic diagram of the design principle of the MO-VLD method;

[0085] Attached Figure 3 This is a schematic diagram comparing the convergence of different methods in 40 physical machine scenarios;

[0086] Attached Figure 4 This is a schematic diagram comparing the convergence of different methods in 120 physical machine scenarios;

[0087] Attached Figure 5 This is a schematic diagram comparing the convergence of different methods in 200 physical machine scenarios;

[0088] Attached Figure 6 It is a schematic diagram of the distribution of Pareto solutions of different methods;

[0089] Attached Figure 7 This is a schematic diagram of the performance comparison of different methods in scenarios with different numbers of physical machines;

[0090] Attached Figure 8 This is a schematic diagram of the impact of different file numbers on the performance of different methods. DETAILED DESCRIPTION

[0091] like Figure 1 As shown, a large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization is shown. The placement method is used to build a Hadoop cluster. The system model of the Hadoop cluster includes multiple racks, each rack is configured with multiple physical machines PMs (Physics Machines, PMs), and each physical machine runs multiple virtual machines VMs (Virtual Machines, VMs); the racks are interconnected through core switches, and the PMs inside the racks are interconnected through switches; the Hadoop cluster transmits data blocks and replicates file copies through switches, thereby achieving data synchronization and load balancing across racks; the optimization goal of the system model of the Hadoop cluster is to minimize the weighted sum of the energy consumption, resource waste and file access delay of the Hadoop cluster, as shown in FIG. Figure 2 As shown, the method adopted by the optimization objective uses a chromosome structure of variable length composed of a virtual machine chromosome and a block chromosome. The chromosome structure is used to describe the locations where virtual machines and data block replicas are placed. The chromosome structure uses crossover and mutation operations to enhance the diversity of solution space exploration of resource configuration in the Hadoop cluster virtual machine placement process.

[0092] In the system model of Hadoop cluster, there are M physical machines in the Hadoop cluster, denoted as PM i (i=1,2,...M), its resource configuration is recorded as a four-tuple Respectively represent the CPU, memory, disk I / O, and network bandwidth of the PM; each PM i Run different numbers of VMs that share the same PM i resources; each VM j The resource usage of (j=1,2,...N) is recorded as a four-tuple The resources allocated to the virtual machine are limited by the resource configuration of the physical machine, which is expressed as In the system model of the Hadoop cluster, the sum of the resources allocated to the virtual machine and the virtualization software overhead must not exceed the resource configuration of the physical machine to avoid excessive consumption of the resources of the physical machine, which is defined by the formula

[0093] Among them, ∑ pmi U pmi Indicates PM i The sum of resource consumption of all virtual machines on the network; Indicates PM i Resource consumption of virtualization software.

[0094] In the system model of Hadoop cluster, VM j The resource consumption of a cannot exceed its allocated resources, which is defined by the formula

[0095]

[0096] In the system model of Hadoop cluster, the storage resource model is as follows: Suppose the file is divided into multiple data blocks B k (k=1,2,...,K) so that its copies are stored on different virtual machines, that is, the file is represented as a connection of blocks, defined by the formula

[0097]

[0098] The file size fSize fu The size of the data block fbSize fu Determines the number of data blocks, which is defined as

[0099] In order to improve data availability and system fault tolerance, each data block is stored in multiple copies on different virtual machines. k ; File f u Each data block There are multiple copies, defined as

[0100]

[0101] Among them, fR fu Represents file f u The number of copies of .

[0102] PM i Multiple copies of the same data block cannot be stored simultaneously to avoid wasting storage resources. This constraint is expressed as

[0103]

[0104] In the system model of the Hadoop cluster, the energy consumption model is as follows:

[0105] Set PM i The energy consumption of PM is affected by CPU, network bandwidth and disk I / O utilization. i The total energy consumption is defined as

[0106]

[0107] in, and They represent the energy consumption of CPU, network bandwidth, and disk I / O respectively;

[0108] Specifically, the CPU energy consumption is defined as follows:

[0109]

[0110] Among them, α i and β i Respectively represent PM i Low CPU load and high CPU load level coefficient; Prh = P max -P idle Indicates the difference between the highest and lowest power of a physical machine; λ i Indicates PM i When the load exceeds the threshold, the CPU energy consumption model will be switched from the low-load CPU energy consumption model to the high-load CPU energy consumption model.

[0111] The network bandwidth energy consumption is defined as

[0112]

[0113] in, represents the network bandwidth utilization, γ i represents the influence coefficient of network bandwidth utilization on energy consumption; disk I / O energy consumption is defined as

[0114]

[0115] in, Indicates disk I / O utilization, δ i Indicates the impact coefficient of disk I / O utilization on energy consumption.

[0116] In the system model of Hadoop cluster, physical machines provide computing, storage and network resources for multiple virtual machines. During the operation of Hadoop cluster, resource waste caused by inefficient or unreasonable resource utilization is expressed as the resource waste related formula in the resource utilization model:

[0117]

[0118] in, Indicates PM iThe resource utilization rate includes CPU, disk bandwidth and network bandwidth utilization; ∈ is a positive coefficient, which is used to avoid unreasonable extreme situations when the standard deviation is zero; Indicates PM i The standard deviation of the utilization of each resource is used to measure the imbalance of resources and is defined as

[0119] in,

[0120] Represents the mean value of each resource utilization, which is defined as

[0121]

[0122] Therefore, PM i The sum of the utilization of all resources is defined as

[0123]

[0124] Assume that in the system model of the Hadoop cluster, the transmission of file data between physical machines needs to pass through multiple switches and physical links. The system model uses switch delay and physical link delay to evaluate the file access delay between virtual machines. The file access delay model of the system model is specifically:

[0125] Assume the switch delay T switch is the processing time required for the data block to pass through the switch, and the file access delay is the physical machine PM where the virtual machine is located. dn Go to the physical machine PM where the target virtual machine is located vm The sum of all switch delays and physical link delays between them is defined as

[0126]

[0127] Where N represents the number of switches in the transmission path, represents the delay of the kth physical link; when there are I data blocks, the average transmission delay of all data blocks is L avg Expressed as

[0128]

[0129] The optimization goal of the Hadoop cluster system model is an NP-hard multi-objective optimization problem. When solving the multi-objective optimization problem, the formal definition of the optimization problem is:

[0130] min(w1Z1+w2Z2+w3Z3) st(1),(2),(6) (17)

[0131] in,

[0132] w1, w2 and w3 represent the weights of the three optimization goals of energy consumption, resource waste and file access delay respectively; the chromosome C of the virtual machine vm is a two-dimensional array, defined as

[0133]

[0134] The first row of each column indicates the physical machine where the virtual machine is placed, and the second row of each column indicates the type of the virtual machine, denoted as VMType. j ∈{1,2,3,4,5}.

[0135] Block chromosome C block is a one-dimensional array, indicating the placement of the data block copy, which is defined as

[0136] C block = {VM1,VM2,...,VM n} (19)

[0137] Among them, each element represents the virtual machine where the data block replica is placed;

[0138] In the initialization phase, C vm Place virtual machines on physical machines in a round-robin manner and randomly assign virtual machine types; C block By generating data block copies in a round-robin and random manner, the distribution of data block copies is ensured to meet the requirements of the Hadoop cluster for the number of copies. The initialization process ensures the diversity of the solution space, so that the MO-VLD method for solving multi-objective optimization problems can fully explore different strategies during the operation phase.

[0139] During the evolution of the chromosome structure, a two-stage crossover and mutation operation is used to generate new solutions and optimize the existing solution set; the two stages are:

[0140] The first stage: single-point crossover of the virtual machine chromosome; randomly select the cutting point p, and exchange the fragments of different parent virtual machine chromosomes to generate the offspring virtual machine chromosome; the process is expressed as

[0141] The second stage: single-point crossover of block chromosomes; randomly select the cutting point q, and exchange the fragments of different parent block chromosomes to generate daughter block chromosomes; the process is expressed as

[0142]

[0143] The mutation operation of the chromosome structure refers to increasing the diversity of solutions by randomly changing the genes in the chromosome;

[0144] In the chromosome structure, the mutation operations on the virtual machine and block chromosomes are defined as follows: For the virtual machine chromosome, in the virtual machine mutation stage, the strategy used to enhance the diversity and optimization ability of the population is as follows: First, the expansion and contraction mutations of the virtual machine adapt to the dynamic resource demand by adjusting the number of virtual machines; the number of virtual machines is first set to The probability of increasing or decreasing provides greater resource configuration flexibility for the solution; then, the virtual machine replacement mutation traverses each element position of the virtual machine chromosome and randomly modifies its value with a probability set to 0.5; on average, this mutation operation will cause half of the virtual machine allocation and template types to change; the method of traversing the virtual machine chromosome includes allocation and template;

[0145] For block chromosomes, a block replacement operation is introduced: by changing the location of data block replicas to optimize their storage location, thereby reducing data transmission latency. Specifically, when a data block replica is located in a certain virtual machine, the block replacement operation will reallocate the replica to other virtual machines with a probability set to 0.5.

[0146] When solving multi-objective optimization problems, the target space is divided into multiple regions, each of which corresponds to a reference point. The reference points are associated and normalized. The reference points are predefined target vectors, which represent potential solutions under different target weights. Specifically, in the evolution of each generation of population, the current solution set will be projected onto the pre-generated reference points and associated according to the distance between the two. Then, the nearest reference point will attract the solution to move in its direction, and then the solution associated with the sparse reference point will be preferentially selected. The selection operator based on the reference point is used to maintain the diversity of the population and ensure the uniform distribution of the solutions, so as to avoid the concentration of solutions in certain target areas and ignore the optimization of other dimensions.

[0147] When solving multi-objective optimization problems, in order to compare different optimization objectives on the same scale, a normalization operation is introduced, specifically: converting individual performance indicators in energy consumption, resource utilization, and file access latency to the same magnitude, thereby eliminating the magnitude differences between different optimization objectives. This process is defined as

[0148]

[0149] Among them, r i represents the weighted target value of the i-th solution; r 1,i 、r 2,i and r 3,i represent energy consumption, resource waste and file access delay respectively; n represents the total number of Pareto solutions.

[0150] The MO-VLD method achieves global optimization of multi-dimensional objectives by combining the reference point association mechanism with the normalization operation.

[0151] The MO-VLD method comprises the following steps:

[0152] Step 1: Construction of chromosomes. When initializing the entire population, generate a combination of decision variables for each individual (chromosome) in the cluster according to the established process; randomly determine the number of virtual machines vmNumber according to the system configuration to ensure that it is not less than the number of physical machines, and then use the double loop and round-robin allocation strategy to generate the data block allocation plan, and add a random shuffle process at the appropriate position to ensure the diversity of solutions. Specifically: through the generateRoundRobin2Dshuffle function, first use generateRoundRobin1D to generate a list of continuous values ​​in a round-robin manner in one dimension, and then divide the list into multiple sub-segments according to the specified block size (ie, the replication factor), randomly shuffle each sub-segment, and then shuffle all sub-segment lists twice, finally forming a two-dimensional structure allocation pattern. After that, the serialize2D function is used to serialize the two-dimensional structure list into a one-dimensional list for subsequent processing and storage;

[0153] Step 2: Generate corresponding sequences for virtual machines vm and virtual machine types vmtype, and use the same round-robin strategy (generateRoundRobin1D) as in step 1 to allocate a virtual machine ID list based on the range of physical machines and a virtual machine type list based on the number of available types (here 4) to vmNumber virtual machines. Then, randomly shuffle the two lists to obtain a decentralized and diverse virtual machine mapping and type allocation scheme in the initial state.

[0154] Step 3: After completing the construction and processing of block, vm, and vmtype, integrate them into a chromosome dictionary, which represents the complete chromosome structure of the individual. Then, by calling the removeEmptyVms function, clean up the generated chromosome (for example, remove unused virtual machine entries) to ensure the validity of the chromosome structure. Finally, add the chromosome to the solution list of the population. At the same time, initialize the dominance relationship and crowding distance and other multi-objective optimization related information for each solution in the population. The corresponding algorithm is expressed in pseudo code as follows:

[0155]

[0156] The corresponding algorithm process is as follows:

[0157] First, the initial maximum time slot T, the maximum number of iterations X, and the population size N are initialized (line 1), and the file list is read from the dataset of the Google cluster (line 2) to simulate the load demand of a real cloud data center;

[0158] In each time slot, the system environment is first initialized, including the configuration information of the physical machine and the virtual machine. In the con_system method, the file size list and the file access rate list are first initialized. Then, according to the file access rate and the predefined resource usage rate, the resource load of each file block is calculated. Specifically, the CPU demand is obtained by multiplying the file access rate by the predefined CPU usage rate. The IO demand is multiplied by a random number based on the CPU demand to simulate the volatility of IO resource usage. The network demand is also multiplied by a random number to simulate the volatility of network resource usage. For each file block and its replica, the calculated CPU, IO and network resource requirements are added to the block_load_list (line 4). In each round of iteration, the population is generated and initialized using the rotation and randomization methods to ensure the diversity of the solution set. During the initialization process, the physical bound load calculation method is used. The system traverses each virtual machine and accumulates the resource requirements of the virtual machine to the corresponding physical machine according to the physical machine index assigned to it, thereby obtaining the total resource usage of each physical machine. To ensure the rationality of resource allocation and the stability of the system, the resource constraint check implemented by the system is as follows: If any resource ( If the usage of a virtual machine or physical machine exceeds its configured capacity (CPU, IO or network), the solution of the corresponding virtual machine placement method will be considered infeasible (line 7); then, a single-point crossover is performed on the virtual machine and block chromosomes, and different mutation operations and block replacement operations are performed to increase population diversity; the virtual machine growth and shrinking methods are used to adjust the number of virtual machines to explore different virtual machine configurations and seek better resource allocation solutions; the virtual machine growth method provides more resource allocation options by increasing the number of virtual machines, which may reduce the load of a single virtual machine; while the virtual machine shrinking method simplifies resource management by reducing the number of virtual machines, but it is necessary to reallocate the data blocks on the deleted virtual machine to other virtual machines to avoid resource overload; when performing the virtual machine shrinking operation, the system first releases the CPU, IO and network resources occupied by the deleted virtual machine; then, the system reallocates the data blocks on the deleted virtual machine to other virtual machines, and through this process affects the CPU, IO and network resource usage of the receiving virtual machine. The system needs to ensure that after the reallocation, the new virtual machine will not affect the overall performance due to resource overload and keep the resource usage within an acceptable range; after the mutation operation, the resource allocation of the solution changes;In order to ensure that the new solution still meets the resource constraints and can accurately evaluate its fitness, the system recalculates the CPU, IO and network resource usage of each virtual machine and physical machine based on the current solution. The method includes traversing all file blocks, accumulating the resource requirements of the virtual machine assigned to each block, and then accumulating the resource usage of the physical machine according to the physical machine assigned to the virtual machine (lines 8-9). Subsequently, the fitness of the offspring is evaluated and merged with the original population, and resource constraints are checked. The system traverses all physical machines to compare whether their CPU, IO and network resource usage exceeds the configured capacity, and then traverses all virtual machines to compare whether their CPU, IO and network resource usage exceeds the configured capacity, so that the system ensures that each mutated solution is reasonable in resource allocation and achieves the optimization goal in overall performance and resource utilization (lines 10-11);

[0159] In order to guide the population to approach the Pareto optimal frontier, pre-generated reference points are introduced, the current solution set is projected onto these reference points, the non-dominated sorting and crowding distance are calculated, and then the next generation population is selected based on the correlation between the two (lines 12-13). To ensure the convergence and global search capabilities of the method, the fitness of the initial population is saved in the initialization phase, and the population quality is improved through multiple iterations. Then, the optimal solution is selected from the merged population as the next generation population, and the current Pareto frontier is saved (lines 14-15). Finally, normalization is used to ensure the balance between different optimization objectives and select the best virtual machine placement strategy (lines 17-18).

[0160] The method for using the virtual machine placement method comprises the following steps:

[0161] Step S1: first deploy a virtual machine resource management module in a large-scale Hadoop cluster environment, and configure initial resource information of physical machines and virtual machines, including CPU, memory, disk I / O, and network bandwidth;

[0162] Step S2: The algorithm used by the MO-VLD method monitors the resource demand changes and task load in the cluster in real time, and loads the corresponding file list or data task to ensure accurate grasp of the actual situation of the cluster, laying the foundation for subsequent optimization;

[0163] Step S3: In resource management, the virtual machine placement method dynamically adjusts the allocation of virtual machines and the placement of data blocks according to the current task load and cluster resource usage. Through intelligent adjustment, the cluster can maximize resource utilization efficiency and reduce energy consumption under different loads;

[0164] Step S4: During the optimization process, the virtual machine placement method continuously analyzes the operation effect of the cluster, and performs strategy feedback and adaptive optimization based on key indicators such as energy consumption, resource utilization, and data access latency to ensure that the cluster system is always in the best operating state.

[0165] Example:

[0166] The algorithm used in this example was carried out on a workstation equipped with an Intel(R) Xeon(R) Silver 4208 CPU and an NVIDIA GeforceGTX 3090 GPU, with a CPU frequency of 2.1 GHz and a memory of 32 GB. The experiment was based on Python 3.9, and a large-scale Hadoop cluster environment was built. Its configuration referred to a typical cloud data center, and a Google cluster dataset was used. The dataset records runtime data obtained from the Borg unit within 7 hours, reflecting the task scheduling and resource utilization in the cloud data center. We initialized the environment according to the arrival of tasks in each time slot to simulate a scenario with dynamically changing loads. Specifically, the settings of the file system are shown in Table 1, and the settings of the physical machine and the virtual machine are shown in Table 2 and Table 3, respectively. In addition, the proposed MO-VLD method is compared with the following five methods.

[0167] (1) The second generation non-dominated sorting genetic algorithm (NSGA-II): offspring are selected through non-dominated sorting and crowding.

[0168] (2) Multi-Objective Particle Swarm Optimization (MOPSO): Utilizes the global search capability of particle swarm to optimize multiple objectives.

[0169] (3) Multi-Objective Simulated Annealing (MOSA): Finds the approximate optimal solution through a temperature-controlled step-by-step annealing process.

[0170] (4) Best-Fit Decreasing (BFD): Prioritizes placing virtual machines on physical machines that best meet their resource requirements.

[0171] (5) First-Fit Decreasing (FFD): The virtual machine is placed on the first physical machine that can meet its resource requirements according to the order of priority.

[0172] In the experiment, the population size of all methods was set to 200, the number of iterations was 300, and the round-robin and random strategies were used for population initialization. MO-VLD and NSGA-II methods used a two-stage crossover strategy for crossover operation. MO-VLD method implemented mutation operation by virtual machine expansion, reduction, allocation and block replacement, while NSGA-II method only used block replacement as mutation operation. The mutation probability was 0.25. The inertia weight, cognitive weight and social weight of MOPSO method were set to 0.5, 1.5 and 1.5 respectively. The initial temperature and cooling rate of MOSA method were 100 and 0.99 respectively.

[0173] Table 1 File system settings

[0174]

[0175] Table 2 Physical machine settings

[0176]

[0177] Table 3 Virtual machine settings

[0178]

[0179] First, this example compares the convergence and Pareto solution set distribution of different methods. The number of files is set to 200, and the number of physical machines is set to 40, 120, and 200, and then the convergence and solution quality of different methods are compared. Figure 3 , Figure 4 and Figure 5 The convergence of different methods in the scenarios of 40, 120 and 200 physical machines was compared respectively. With the increase of the number of iterations, all methods gradually converged. The MO-VLD method can achieve fast convergence in a fewer number of iterations, and its final optimization effect is better than other methods. This is because the MO-VLD method adopts a dual-chromosome structure and a two-stage crossover and mutation operation, which increases the diversity of solutions and avoids the method from falling into the local optimum. At the same time, the MO-VLD method introduces a reference point association mechanism, which can better balance multiple optimization objectives and ensure that each optimization objective can be effectively optimized. In contrast, although the NSGA-II method has a faster convergence speed in the early stages of iteration, its crossover and mutation operation design is too simple, resulting in slightly inferior optimization effects in the later stages. The MOPSO method is prone to falling into local optimality, and although the MOSA method is easy to obtain better solutions in the early iterations, its annealing process leads to slow convergence. Specifically, Figure 3As shown in Figure 2, when the number of physical machines is 40, compared with NSGA-II, MOPSO and MOSA, the proposed MO-VLD method reduces energy consumption by about 3.12%, 7.9% and 4.3%, improves resource utilization by about 1.4%, 7.6% and 9%, and reduces file access latency by about 0.7%, 17.2% and 22.2%, respectively. Figure 4 As shown in the figure, when the number of physical machines is 120, compared with NSGA-II, MOPSO and MOSA methods, the proposed MO-VLD method reduces energy consumption by about 4.47%, 6.70% and 9.3%, respectively, and improves resource utilization by 1.

[0180] The performance of the two platforms increased by 5.14%, 5.25%, and 13.04%, respectively, and the file access latency decreased by 1.37%, 4.17%, and 6.91%, respectively. Figure 5 As shown in the figure, when the number of physical machines is 200, compared with NSGA-II, MOPSO and MOSA methods, the proposed MO-VLD method reduces energy consumption by about 1.0%, 1.8% and 5.8%, respectively, improves resource utilization by about 0.9%, 5.9% and 7.4%, respectively, and reduces file access latency by about 1.3%, 8.5% and 10.3%, respectively.

[0181] Figure 6 The distribution of Pareto solution sets of different methods is compared. Among these methods, the performance of MOSA, MOPSO, NSGA-II and the proposed MO-VLD method increases in turn. Compared with the other three methods, the Pareto solution set of the MO-VLD method is more inclined to the origin of the three-dimensional coordinate axis, which shows that the quality of the solution obtained by the MO-VLD method is higher. This is because the MO-VLD method introduces crossover and mutation operations, so its global search ability is better than other methods. As the number of physical machines increases, the number of Pareto solution sets also increases, which shows that as the complexity of the scene increases, the number of potential feasible solutions will also increase.

[0182] Furthermore, this example compares the performance of different methods under different physical machine number scenarios. The number of physical machines is set to 40, 80, 120, 160 and 200, and the number of virtual machines is twice the number of physical machines. When the number of physical machines is less than 100, the number of racks is 5; otherwise, the number of racks is 8. The above settings are used to simulate the configuration of a large-scale cluster. The present invention normalizes and weights the three performance indicators of energy consumption, resource waste and file access latency, and then compares the performance of different methods. Figure 7As shown in the figure, the weighted objective value of the proposed MO-VLD method is the lowest. This is because the MO-VLD method adopts a dual-chromosome structure, which allows the simultaneous optimization of virtual machine and data block replica placement, greatly reducing cross-rack data transmission and unbalanced use of physical machine resources. At the same time, the two-stage crossover and mutation operations in the MO-VLD method also enhance its ability to explore the solution space. Therefore, the MO-VLD method performs well in dealing with multi-objective optimization problems, and it can always maintain a good and stable optimization effect regardless of how the number of physical machines changes. The weighted objective value of the NSGA-II method is only about half of that of the FFD and BFD methods, and is about 0.05 lower than that of the MOPSO and MOSA methods. Although the NSGA-II method performs relatively stably in various scenarios, due to its relatively simple crossover and mutation operations, it is difficult to effectively balance multiple optimization objectives in large-scale cluster scenarios, resulting in its performance slightly inferior to the MO-VLD method, especially when the number of physical machines increases to 200. The performance of the MOPSO and MOSA methods is always inferior to the proposed MO-VLD method. This is because the MOPSO method is prone to fall into local optimality during the particle swarm search process and is difficult to achieve global equilibrium in multi-objective optimization problems. Although the MOSA method has certain global search capabilities, its annealing process causes excessive operating overhead and performs poorly when facing large-scale clusters. FFD and BFD are heuristic algorithms that seriously lack global optimization capabilities when facing large-scale clusters, so the optimization effect is the worst.

[0183] Finally, this example evaluates the impact of different file numbers on the performance of different methods. The number of physical machines is set to 200, and the number of files is set to 40, 80, 120, 160, and 200. Figure 8 As shown in the figure, the proposed MO-VLD method can achieve the best performance in different file number scenarios. In terms of energy consumption, the energy consumption of MO-VLD is lower than that of other methods in all file number scenarios, and as the number of files increases, the MO-VLD method can maintain low energy consumption. This is because the dual chromosome structure in the MO-VLD method can achieve

[0184] The virtual machines and data blocks are optimally placed to avoid resource waste or overuse. Compared with NSGA-II, MOPSO, MOSA, FFD and BFD methods, the MO-VLD method can achieve an average performance improvement of about 0.4%, 5.0%, 5.6%, 23.6% and 26.2% in different scenarios. As the number of files increases, the resource waste of all methods shows a downward trend. Compared with other methods, the MO-VLD method can maintain a stable low resource waste in different file number scenarios. This is because the MO-VLD method adopts dynamic resource scheduling and block replacement mechanism, which can be adjusted in time when resource demand changes. Compared with NSGA-II, MOPSO, MOSA, FFD and BFD methods, the MO-VLD method achieves an average performance improvement of about 16.2%, 26.0%, 28.9%, 35.1% and 38.4% in different scenarios. In terms of file access latency, although the performance of different methods is not much different, the MO-VLD method still shows good performance. This shows that the MO-VLD method can effectively reduce data transmission time and improve the file access efficiency of the cluster. Specifically, compared with NSGA-II, MOPSO, MOSA, FFD and BFD methods, the MO-VLD method can achieve an average performance improvement of about 1.8%, 8.9%, 8.5%, 7.5% and 6.2% in different scenarios.

Claims

1. A large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization, characterized by: The placement method is used to form a Hadoop cluster, wherein the system model of the Hadoop cluster includes multiple racks, each rack is configured with multiple physical machines PMs, and each physical machine runs multiple virtual machines VMs; the racks are interconnected through core switches, and the PMs inside the racks are interconnected through switches; the Hadoop cluster transmits data blocks and replicates file copies through the switches, thereby realizing data synchronization and load balancing across racks; the optimization goal of the system model of the Hadoop cluster is to minimize the weighted sum of energy consumption, resource waste and file access delay of the Hadoop cluster, and the method adopted by the optimization goal uses a variable-length chromosome structure composed of virtual machine chromosomes and block chromosomes, the chromosome structure is used to express the location of placing virtual machines and data block copies, and the chromosome structure uses crossover and mutation operations to enhance the diversity of solution space exploration of resource configuration in the Hadoop cluster virtual machine placement process.

2. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 1 is characterized in that: In the system model of Hadoop cluster, there are M physical machines in the Hadoop cluster, denoted as PM i (i=1,2,...M), its resource configuration is recorded as a four-tuple Respectively represent the CPU, memory, disk I / O, and network bandwidth of the PM; each PM i Run different numbers of VMs that share the same PM i resources; each VM j The resource usage of (j=1,2,...N) is recorded as a four-tuple The resources allocated to the virtual machine are limited by the resource configuration of the physical machine, which is expressed as In the system model of the Hadoop cluster, the sum of the resources allocated to the virtual machine and the virtualization software overhead must not exceed the resource configuration of the physical machine to avoid excessive consumption of the resources of the physical machine, which is defined by the formula Among them, ∑ pmi U pmi Indicates PM i The sum of resource consumption of all virtual machines on the network; Indicates PM i Resource consumption of virtualization software; In the system model of Hadoop cluster, VM j The resource consumption of a cannot exceed its allocated resources, which is defined by the formula 3. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 2 is characterized in that: In the system model of Hadoop cluster, the storage resource model is as follows: Suppose the file is divided into multiple data blocks B k (k=1,2,...,K) so that its copies are stored on different virtual machines, that is, the file is represented as a connection of blocks, defined by the formula The file size fSize fu The size of the data block fbSize fu Determines the number of data blocks, which is defined as In order to improve data availability and system fault tolerance, each data block is stored in multiple copies on different virtual machines. k ; File f u Each data block There are multiple copies, defined as Among them, fR fu Represents file f u The number of copies; PM i Multiple copies of the same data block cannot be stored simultaneously to avoid wasting storage resources. This constraint is expressed as 4. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 3 is characterized in that: In the system model of the Hadoop cluster, the energy consumption model is as follows: Set PM i The energy consumption of PM is affected by CPU, network bandwidth and disk I / O utilization. i The total energy consumption is defined as in, and They represent the energy consumption of CPU, network bandwidth, and disk I / O respectively; Specifically, the CPU energy consumption is defined as follows: Among them, α i and β i Respectively represent PM i Low CPU load and high CPU load level coefficient; Prh = P max -P idle Indicates the difference between the highest and lowest power of a physical machine; λ i Indicates PM i When the load exceeds the threshold, the CPU energy consumption model will be switched from the low-load CPU energy consumption model to the high-load CPU energy consumption model. The network bandwidth energy consumption is defined as in, represents the network bandwidth utilization, γ i represents the influence coefficient of network bandwidth utilization on energy consumption; disk I / O energy consumption is defined as in, Indicates disk I / O utilization, δ i Indicates the impact coefficient of disk I / O utilization on energy consumption.

5. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 4 is characterized in that: In the system model of Hadoop cluster, physical machines provide computing, storage and network resources for multiple virtual machines. During the operation of Hadoop cluster, resource waste caused by inefficient or unreasonable resource utilization is expressed as the resource waste related formula in the resource utilization model: in, Indicates PM i The resource utilization rate includes CPU, disk bandwidth and network bandwidth utilization; ∈ is a positive coefficient, which is used to avoid unreasonable extreme situations when the standard deviation is zero; Indicates PM i The standard deviation of the utilization of each resource is used to measure the imbalance of resources and is defined as in, Represents the mean value of each resource utilization, which is defined as Therefore, PM i The sum of the utilization of all resources is defined as 6. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 5 is characterized in that: Assume that in the system model of the Hadoop cluster, the transmission of file data between physical machines needs to pass through multiple switches and physical links. The system model uses switch delay and physical link delay to evaluate the file access delay between virtual machines. The file access delay model of the system model is as follows: Assume that the switch delay T switch is the processing time required for the data block to pass through the switch, and the file access delay is the physical machine PM where the virtual machine is located. dn Go to the physical machine PM where the target virtual machine is located vm The sum of all switch delays and physical link delays between them is defined as Where N represents the number of switches in the transmission path, represents the delay of the kth physical link; When there are I data blocks, the average transmission delay of all data blocks is L avg Expressed as 7. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 6 is characterized in that: The optimization goal of the Hadoop cluster system model is an NP-hard multi-objective optimization problem. When solving the multi-objective optimization problem, the formal definition of the optimization problem is min(w1Z1+w2Z2+w3Z3)st(1),(2),(6) (17) in, w1, w2, and w3 represent the weights of the three optimization objectives of energy consumption, resource waste, and file access latency, respectively; VM Chromosome C vm is a two-dimensional array, defined as The first row of each column indicates the physical machine where the virtual machine is placed, and the second row of each column indicates the type of the virtual machine, denoted as VMType. j ∈{1,2,3,4,5}; Block chromosome C block It is a one-dimensional array, indicating the placement of the data block copy, which is defined as C block = {VM1,VM2,...,VM n } (19) Among them, each element represents the virtual machine where the data block replica is placed; In the initialization phase, C vm Place virtual machines on physical machines in a round-robin manner and randomly assign virtual machine types; C block By rotating and randomly generating, the distribution of data block copies is ensured to meet the requirements of the Hadoop cluster for the number of copies. The initialization process ensures the diversity of the solution space, so that the MO-VLD method for solving multi-objective optimization problems can fully explore different strategies during the operation phase. During the evolution of the chromosome structure, a two-stage crossover and mutation operation is used to generate new solutions and optimize the existing solution set. The two stages are: The first stage: single-point crossover of the virtual machine chromosome; randomly select the cutting point p, and exchange the fragments of different parent virtual machine chromosomes to generate the offspring virtual machine chromosome; the process is expressed as The second stage: single-point crossover of block chromosomes; randomly select the cutting point q, and exchange the fragments of different parent block chromosomes to generate daughter block chromosomes; the process is expressed as The mutation operation of the chromosome structure refers to increasing the diversity of solutions by randomly changing the genes in the chromosome; In the chromosome structure, the mutation operations on the virtual machine and block chromosomes are defined as follows: For the virtual machine chromosome, in the virtual machine mutation stage, the strategy used to enhance the diversity and optimization ability of the population is as follows: First, the expansion and contraction mutation of the virtual machine adapts to the dynamic resource demand by adjusting the number of virtual machines; wherein the number of virtual machines is first increased or decreased with a probability set to M, providing greater resource configuration flexibility for the solution; then, the virtual machine replacement mutation traverses each element position of the virtual machine chromosome and randomly modifies its value with a probability set to N; the methods for traversing the virtual machine chromosome include allocation and template; For block chromosomes, a block replacement operation is introduced: by changing the location of data block copies, the storage location is optimized, thereby reducing data transmission delay; specifically, when a data block copy is located in a virtual machine, the block replacement operation will reallocate the copy to other virtual machines with a probability set to N.

8. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 7 is characterized in that: When solving multi-objective optimization problems, the target space is divided into multiple regions, each of which corresponds to a reference point. The reference points are associated and normalized. The reference points are predefined target vectors, which represent potential solutions under different target weights. Specifically, in the evolution of each generation of population, the current solution set will be projected onto the pre-generated reference points and associated according to the distance between the two. Then, the nearest reference point will attract the solution to move in its direction, and then the solution associated with the sparse reference point will be preferentially selected. The selection operator based on the reference point is used to maintain the diversity of the population and ensure the uniform distribution of the solutions, so as to avoid the concentration of solutions in certain target areas and ignore the optimization of other dimensions. When solving multi-objective optimization problems, in order to compare different optimization objectives on the same scale, a normalization operation is introduced. Specifically, the individual performance indicators of energy consumption, resource utilization, and file access latency are converted to the same magnitude, thereby eliminating the magnitude differences between different optimization objectives. This process is defined as Among them, r i represents the weighted target value of the i-th solution; r 1,i 、r 2,i and r 3,i They represent energy consumption, resource waste and file access delay respectively; n represents the total number of Pareto solution sets; The MO-VLD method achieves global optimization of multi-dimensional objectives by combining the reference point association mechanism with the normalization operation.

9. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 8, characterized in that: MO-VLD method The steps include: Step 1: Chromosome construction. When initializing the entire population, generate a combination of decision variables for each individual in the cluster, i.e., chromosome, according to the established process; randomly determine the number of virtual machines vmNumber according to the system configuration to ensure that it is not less than the number of physical machines, and then use the double loop and round-robin allocation strategy to generate the allocation plan for the data block block, and add a random shuffle process at an appropriate position to ensure the diversity of solutions. Specifically, through the generateRoundRobin2Dshuffle function, first use generateRoundRobin1D to generate a list of continuous values ​​in a round-robin manner on one dimension, and then divide the list into multiple sub-segments according to the specified block size, i.e., the replication factor, and randomly shuffle each sub-segment. Then, shuffle all sub-segment lists twice, and finally form a two-dimensional structure allocation pattern; thereafter, through the serialize2D function, serialize the two-dimensional structure list into a one-dimensional list for subsequent processing and storage; Step 2: Generate corresponding sequences for virtual machines vm and virtual machine types vmtype, and use the same round-robin strategy generateRoundRobin1D as in step 1 to allocate a virtual machine ID list based on the range of physical machine numbers and a virtual machine type list based on the number of available types to vmNumber virtual machines, and then randomly shuffle the two lists to obtain a decentralized and diverse virtual machine mapping and type allocation scheme in the initial state; Step 3: After completing the construction and processing of block, vm, and vmtype, integrate them into a chromosome dictionary, which represents the complete chromosome structure of the individual. Then, the generated chromosome is cleaned up by calling the removeEmptyVms function to remove unused virtual machine entries to ensure the validity of the chromosome structure. Finally, the chromosome is added to the solution list of the population. At the same time, the multi-objective optimization related information of the dominance relationship and crowding distance is initialized for each solution in the population. The corresponding algorithm flow is as follows: First, the initial maximum time slot T, maximum number of iterations X, and population size N are initialized, and the file list is read from the dataset of the Google cluster to simulate the load demand of a real cloud data center; In each time slot, the system environment is first initialized, including the configuration information of the physical machine and the virtual machine. In the con_system method, the file size list and the file access rate list are first initialized. Then, the resource load of each file block is calculated according to the file access rate and the predefined resource usage rate. Specifically, the CPU demand is obtained by multiplying the file access rate by the predefined CPU usage rate. The IO demand is multiplied by a random number based on the CPU demand to simulate the volatility of IO resource usage. The network demand also simulates the volatility of network resource usage by multiplying the CPU demand by a random number. For each file block and its replica, the calculated CPU, IO and network resource requirements are added to block_load_list. In each round of iteration, the population is generated and initialized using the rotation and randomization methods to ensure the diversity of the solution set. During the initialization process, the system traverses each virtual machine using the physical load calculation method, and according to the physical machine index assigned to it, the resource requirements of the virtual machine are accumulated to the corresponding physical machine, thereby obtaining the total resource usage of each physical machine. To ensure the rationality of resource allocation and the stability of the system, the resource constraint check implemented by the system is as follows: If the usage of any resource in terms of CPU, IO or network on a virtual machine or physical machine exceeds its configured capacity, the solution of the corresponding virtual machine placement method will be deemed infeasible. Then, a single-point crossover is performed on the virtual machine and block chromosomes, and different mutation operations and block replacement operations are performed to increase the diversity of the population. The virtual machine growth and shrinkage methods are used to adjust the number of virtual machines to explore different virtual machine configurations and seek a better resource allocation solution. The VM growth method provides more resource allocation options by increasing the number of VMs, thereby reducing the load on a single VM; The virtual machine shrinking method simplifies resource management by reducing the number of virtual machines. It needs to reallocate data blocks on the deleted virtual machine to other virtual machines to avoid resource overload. When performing a virtual machine shrinking operation, the system first releases the CPU, IO, and network resources occupied by the deleted virtual machine. Subsequently, the system reallocates data blocks on the deleted virtual machine to other virtual machines. This process affects the CPU, IO, and network resource usage of the receiving virtual machine. The system needs to ensure that after the reallocation, the new virtual machine will not affect the overall performance due to resource overload and keep resource usage within an acceptable range. After the mutation operation, the resource allocation of the solution changes. In order to ensure that the new solution still meets the resource constraints and can accurately evaluate its fitness ,Based on the current solution, the system recalculates the CPU, IO and network resource usage of each virtual machine and physical machine. The method includes traversing all file blocks, accumulating the resource requirements of the virtual machine assigned to each block, and then accumulating the resource usage of the physical machine according to the physical machine assigned to the virtual machine. Subsequently, the fitness of the offspring is evaluated and merged with the original population, and resource constraints are checked. The system traverses all physical machines to compare whether their CPU, IO and network resource usage exceeds the configured capacity, and then traverses all virtual machines to compare whether their CPU, IO and network resource usage exceeds the configured capacity, so that the system ensures that each mutated solution is reasonable in resource allocation and achieves the optimization goals in terms of overall performance and resource utilization; In order to guide the population to approach the Pareto optimal front, pre-generated reference points are introduced, and the current solution set is projected onto these reference points. The non-dominated sorting and crowding distance are calculated, and then the next generation population is selected according to the correlation between the two. To ensure the convergence and global search ability of the method, the fitness of the initial population is saved in the initialization stage, and the population quality is improved through multiple iterations. Then, the optimal solution is selected from the merged population as the next generation population, and the current Pareto front is saved. Finally, normalization is used to ensure the balance between different optimization objectives and select the best virtual machine placement strategy.

10. The large-scale Hadoop cluster virtual machine placement method based on multi-objective optimization according to claim 9, characterized in that: The method for using the virtual machine placement method comprises the following steps: Step S1: first deploy a virtual machine resource management module in a large-scale Hadoop cluster environment, and configure initial resource information of physical machines and virtual machines, including CPU, memory, disk I / O, and network bandwidth; Step S2: The algorithm used by the MO-VLD method monitors the resource demand changes and task load in the cluster in real time, and loads the corresponding file list or data task to ensure accurate grasp of the actual situation of the cluster, laying the foundation for subsequent optimization; Step S3: In resource management, the virtual machine placement method dynamically adjusts the allocation of virtual machines and the placement of data blocks according to the current task load and cluster resource usage; through intelligent adjustment, the cluster maximizes resource utilization efficiency and reduces energy consumption under different loads; Step S4: During the optimization process, the virtual machine placement method continuously analyzes the operation effect of the cluster, and performs strategy feedback and adaptive optimization based on key indicators of energy consumption, resource utilization, and data access latency to ensure that the cluster system is always in the best operating state.

Images