A method for placing virtual machines in a large-scale Hadoop cluster based on multi-objective optimization.

Abstract

This invention proposes a multi-objective optimization-based method for placing virtual machines (VMs) in a large-scale Hadoop cluster. The Hadoop cluster system model comprises multiple racks, each containing multiple physical machines (PMs), and each PM running multiple virtual machines (VMs). The Hadoop cluster uses switches for data block transmission and file copy replication, thereby achieving cross-rack data synchronization and load balancing. The optimization objective of the Hadoop cluster system model is to minimize the weighted sum of energy consumption, resource waste, and file access latency. The optimization method employs a variable-length chromosome structure composed of VM chromosomes and block chromosomes. This chromosome structure describes the placement of VMs and data block copies. The chromosome structure uses crossover and mutation operations to enhance the diversity of the solution space exploration for resource configuration during VM placement. This invention fully considers the dynamic changes in resource requirements during Hadoop cluster operation, ensuring that the Hadoop cluster maintains good performance under high load conditions.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method for placing virtual machines in a large-scale Hadoop cluster based on multi-objective optimization. Background Technology

[0002] With the rapid development of cloud computing, virtualization technology has become a core support for cloud data centers. By accessing cloud computing services, users can access and utilize resources (such as CPU / GPU, memory, and storage) on demand. This on-demand service significantly improves the resource utilization of cloud data centers while reducing their operating costs. Therefore, more and more users are uploading their computing tasks to the cloud to meet their ever-increasing data processing needs.

[0003] As a classic distributed framework, Hadoop's excellent scalability and flexibility have led to its widespread adoption in cloud environments. One of Hadoop's core technologies, the Hadoop Distributed File System (HDFS), divides files into multiple data blocks and stores them across multiple data nodes, ensuring high availability and fault tolerance. Another core Hadoop technology, MapReduce, schedules computational tasks to the nodes where data blocks reside to maximize data locality and reduce data transmission latency. However, with the continuous growth of data volume and cluster size, how to efficiently optimize virtual machine placement to improve cluster resource utilization and task processing efficiency has become a core issue in Hadoop cluster performance optimization. Virtual machine placement strategies directly affect key performance indicators such as Hadoop cluster energy consumption, resource utilization, and file access latency. Therefore, how to efficiently manage resources in large-scale, dynamic Hadoop clusters has become an open research hotspot.

[0004] Traditional virtual machine placement mostly employs strategies based on genetic algorithms, which demonstrate good performance with small-scale clusters but exhibit limitations when facing large-scale clusters and variable loads, failing to adapt well to dynamically changing resource demands. To address this issue, some studies have proposed multi-objective optimization algorithms, such as Multi-Objective Particle Swarm Optimization (MOPSO) and Multi-Objective Simulated Annealing (MOSA). Specifically, the MOPSO algorithm performs a global search by simulating swarm behavior, but it is prone to getting trapped in local optima in complex environments and is sensitive to hyperparameters, making the strategy optimization process difficult to control. The MOSA algorithm finds the global optimum by simulating energy changes during physical annealing, avoiding local optima, but its convergence speed is slow and it incurs excessive system overhead when handling multi-objective optimization problems. Furthermore, these algorithms cannot effectively handle multi-dimensional heterogeneous resources in virtual machine placement problems. To address the aforementioned key challenges, this invention proposes a novel Multi-objective Optimization method with Variable Length Double Chromosome (MO-VLD) for placing virtual machines in large-scale Hadoop clusters. This method considers energy consumption, resource utilization, and file access latency as multiple optimization objectives. In particular, the proposed system model fully considers the dynamic changes in resource requirements to ensure that the Hadoop cluster maintains good performance under high load conditions. Summary of the Invention

[0005] This invention proposes a method for placing virtual machines in a large-scale Hadoop cluster based on multi-objective optimization. It considers energy consumption, resource utilization, and file access latency as multiple optimization objectives. The system model used in the method fully considers the dynamic changes in resource requirements to ensure that the Hadoop cluster can maintain good performance under high load.

[0006] The present invention adopts the following technical solution.

[0007] A multi-objective optimization-based method for placing virtual machines in a large-scale Hadoop cluster is proposed. This method is used to build a Hadoop cluster, whose system model includes multiple racks. Each rack contains multiple physical machines (PMs), and each PM runs multiple virtual machines (VMs). The racks are interconnected via a core switch, and the PMs within each rack are interconnected via switches. The Hadoop cluster uses switches for data block transfer and file copy replication, achieving cross-rack data synchronization and load balancing. The optimization objective of the Hadoop cluster system model is to minimize the weighted sum of energy consumption, resource waste, and file access latency. The optimization method employs a variable-length chromosome structure composed of virtual machine chromosomes and block chromosomes. This chromosome structure describes the placement locations of virtual machines and data block copies. The chromosome structure uses crossover and mutation operations to enhance the diversity of the solution space exploration in the resource configuration process during virtual machine placement.

[0008] In the system model of a Hadoop cluster, there are M physical machines in the Hadoop cluster, denoted as PM. i (i = 1, 2, ..., M), its resource allocation is denoted as a quadruple. These represent the CPU, memory, disk I / O, and network bandwidth of the PM, respectively; each PM i Running different numbers of VMs, which share a PM i Resources; per VM j The resource usage of (j = 1, 2, ..., N) is denoted as a quadruple. The resources allocated to a virtual machine are limited by the resource configuration of the physical machine, which is represented as follows: In the system model of the Hadoop cluster, the sum of the resources allocated to the virtual machines and the virtualization software overhead must not exceed the resource configuration of the physical machines to avoid excessive consumption of physical machine resources, as defined by the formula.

[0009] Where, ∑ pmi U pmi Indicates PM i The sum of resource consumption of all virtual machines; Indicates PM i Resource consumption of virtualization software.

[0010] In the system model of a Hadoop cluster, VM j Resource consumption cannot exceed the resources allocated to it, as defined by the formula.

[0011]

[0012] In the system model of a Hadoop cluster, the storage resource model is specifically as follows: Assume a file is split into multiple data blocks B. k (k = 1, 2, ..., K), so that its copies are stored on different virtual machines, i.e., the file is represented as a concatenation of blocks, defined by the formula...

[0013]

[0014] File size fSize fu With the size of the data block fbSize fu The number of data blocks is determined by the following definition:

[0015] To improve data availability and system fault tolerance, each data block is stored on multiple replicas on different virtual machines. k ; file f u Each data block There are multiple copies, which are defined as follows:

[0016]

[0017] Among them, fR fu Represents file f u The number of copies.

[0018] PM i Multiple copies of the same data block cannot be stored simultaneously to avoid wasting storage resources; this constraint is expressed as follows:

[0019]

[0020] In the system model of a Hadoop cluster, the energy consumption model is as follows:

[0021] Let PM i The energy consumption of PM is affected by CPU, network bandwidth, and disk I / O utilization. i Total energy consumption is defined by the formula

[0022]

[0023] in, and These represent the energy consumption of CPU, network bandwidth, and disk I / O, respectively.

[0024] Specifically, CPU energy consumption is defined by the formula as follows:

[0025]

[0026] Where, αi and β i They represent PM i Low CPU load and high CPU load level coefficients; Prh = P max -P idle λ represents the difference between the highest and lowest power of the physical machine. i Indicates PM i The load threshold is set, and when the load exceeds the threshold, the CPU will switch from the low load CPU power consumption model to the high load CPU power consumption model.

[0027] Network bandwidth energy consumption is defined as

[0028]

[0029] in, γ represents network bandwidth utilization. i This represents the impact coefficient of network bandwidth utilization on energy consumption; disk I / O energy consumption is defined as...

[0030]

[0031] in, This represents disk I / O utilization, δ i This represents the coefficient indicating the impact of disk I / O utilization on energy consumption.

[0032] In the system model of a Hadoop cluster, physical machines provide computing, storage, and network resources for multiple virtual machines. During the operation of a Hadoop cluster, resource waste caused by inefficient or unreasonable resource utilization is expressed by the relevant formulas in the resource utilization model.

[0033]

[0034] in, Indicates PM i Resource utilization, including CPU, disk bandwidth and network bandwidth utilization; ∈ is a positive coefficient used to avoid unreasonable extreme cases when the standard deviation is zero; Indicates PM i The standard deviation of the utilization rates of various resources is used to measure resource imbalance, and is defined by the formula:

[0035] in,

[0036] The average utilization rate of each resource is defined as follows:

[0037]

[0038] Therefore, PMi The sum of the utilization rates of all resources is defined as

[0039]

[0040] In a Hadoop cluster system model, file data transmission between physical machines requires multiple switches and physical links. The system model uses switch latency and physical link latency to evaluate file access latency between virtual machines. The specific file access latency model of the system model is as follows:

[0041] Let the switch delay T switch If the processing time required for a data block to pass through the switch is given, then the file access latency is the PM of the physical machine where the virtual machine resides. dn To the physical machine PM where the target virtual machine is located vm The sum of all switch delays and physical link delays between them, expressed in the formula table, is defined as follows:

[0042]

[0043] Where N represents the number of switches in the transmission path. This represents the delay of the k-th physical link; when there are I data blocks, the average transmission delay L of all data blocks is... avg Represented as

[0044]

[0045] The optimization objective of the Hadoop cluster system model is an NP-hard multi-objective optimization problem. When solving this multi-objective optimization problem, its formal definition is...

[0046] min(w1Z1+w2Z2+w3Z3)st(1),(2),(6)(17)

[0047] in, w1, w2, and w3 represent the weights of the three optimization objectives: energy consumption, resource waste, and file access latency, respectively; Virtual Machine Chromosome C vm It is a two-dimensional array, defined as follows:

[0048]

[0049] In this table, the first row of each column represents the physical machine on which the virtual machine is placed, and the second row of each column represents the type of virtual machine, denoted as VMType. j ∈{1,2,3,4,5}.

[0050] Block chromosome C block This is a one-dimensional array representing the placement location of data block replicas, defined as follows:

[0051] C block ={VM1, VM2, ... , VM n} (19)

[0052] Each element represents the virtual machine where the data block replica is placed;

[0053] During the initialization phase, C vm Virtual machines are placed on physical machines using a round-robin method, and the virtual machine type is randomly assigned; C block By generating data block replicas in a round-robin and random manner, the distribution of data block replicas is ensured to meet the requirements of the Hadoop cluster for the number of replicas. The process in the initialization phase guarantees the diversity in the solution space, so that the MO-VLD method for solving multi-objective optimization problems can fully explore different strategies during the runtime phase.

[0054] During evolution, chromosome structure utilizes a two-stage crossover and mutation process to generate new solutions and optimize the existing solution set; the two stages are:

[0055] Phase 1: Single-point crossover of virtual machine chromosomes; randomly selecting a cut point p, exchanging fragments of different parent virtual machine chromosomes to generate child virtual machine chromosomes; this process is expressed by the formula:

[0056] The second stage involves single-point crossover of the chromosome blocks; a random cutting point q is selected, and segments from different parent chromosome blocks are exchanged to generate daughter chromosome blocks; this process is expressed by the following formula:

[0057]

[0058] The mutation operation of the chromosome structure refers to increasing the diversity of solutions by randomly changing the genes in the chromosome;

[0059] In the chromosome structure, the mutation operations on virtual machines and block chromosomes are defined as follows: For virtual machine chromosomes, during the virtual machine mutation phase, the strategy to enhance population diversity and optimization capabilities is as follows: First, the expansion and contraction mutations of virtual machines adapt to dynamic resource demands by adjusting the number of virtual machines; whereby the number of virtual machines is initially set to... The probability of increasing or decreasing provides greater resource configuration flexibility for the solution; then, the virtual machine replacement mutation traverses the position of each element of the virtual machine chromosome and randomly modifies its value with a probability set to 0.5; on average, this mutation operation will cause half of the virtual machine allocations and template types to change; the method of traversing the virtual machine chromosome includes allocations and templates;

[0060] For block chromosomes, a block replacement operation is introduced: by changing the location of data block replicas, their storage location is optimized, thereby reducing data transfer latency. Specifically, when a data block replica is located in a virtual machine, the block replacement operation will reallocate the replica to another virtual machine with a probability set to 0.5.

[0061] When solving multi-objective optimization problems, the objective space is divided into multiple regions, each corresponding to a reference point. Reference points are associated and normalized. Each reference point is a predefined objective vector representing a potential solution under different objective weights. Specifically, during the evolution of each generation of the population, the current solution set is projected onto pre-generated reference points and associated based on the distance between them. The nearest reference point attracts solutions to move towards it, thus prioritizing solutions associated with sparse reference points. This reference-point-based selection operator maintains population diversity and ensures a uniform distribution of solutions, preventing solutions from concentrating in certain objective regions and neglecting optimization in other dimensions.

[0062] When solving multi-objective optimization problems, a normalization operation is introduced to compare different optimization objectives on the same scale. Specifically, this involves transforming the performance indicators of individuals in terms of energy consumption, resource utilization, and file access latency into the same order of magnitude, thereby eliminating the order-of-magnitude differences between different optimization objectives. This process is defined as follows:

[0063]

[0064] Where, r i r represents the weighted objective value of the i-th scheme; 1,i r 2,i and r 3,i represents energy consumption, resource waste, and file access latency, respectively; n represents the total number of Pareto solutions.

[0065] The MO-VLD method achieves global optimization of multidimensional objectives by combining a reference point association mechanism with normalization operations.

[0066] The MO-VLD method includes the following steps;

[0067] Step 1: Chromosome Construction. During population initialization, a set of decision variables is generated for each individual (chromosome) according to a predetermined process. The number of virtual machines (vmNumber) is randomly determined based on system configuration, ensuring it is no less than the number of physical machines. Then, a double-loop and round-robin allocation strategy is used to generate data block allocation schemes, with random shuffling added at appropriate locations to guarantee solution diversity. Specifically, the `generateRoundRobin2Dshuffle` function first generates a continuous list of values ​​in a one-dimensional round-robin manner using `generateRoundRobin1D`. This list is then divided into multiple sub-fragments according to a specified block size (i.e., replication factor). Each sub-fragment is randomly shuffled, and all sub-fragment lists are shuffled a second time, ultimately forming a two-dimensional allocation pattern. Subsequently, the `serialize2D` function serializes this two-dimensional list into a one-dimensional list for subsequent processing and storage.

[0068] Step 2: Generate corresponding sequences for virtual machines (vm) and virtual machine types (vmtype). Using the same round-robin strategy as in Step 1 (generateRoundRobin1D), assign a list of virtual machine IDs based on the number of physical machines and a list of virtual machine types based on the number of available types (4 in this case) to the vmNumber virtual machines. Then, randomly shuffle these two lists to obtain a dispersed and diverse virtual machine mapping and type allocation scheme in the initial state.

[0069] Step 3: After constructing and processing the block, vm, and vmtype, they are integrated and stored in a chromosome dictionary, representing the complete chromosome structure of the individual. The generated chromosome is then cleaned by calling the removeEmptyVms function (e.g., removing unused virtual machine entries) to ensure the validity of the chromosome structure. Finally, the chromosome is added to the solution list of the population. Simultaneously, multi-objective optimization information such as dominance relationships and crowding distances is initialized for each solution in the population. The corresponding algorithm is expressed in pseudocode as follows:

[0070]

[0071] The specific algorithm flow is as follows:

[0072] First, the initial maximum time slot T, maximum number of iterations X, and population size N are initialized (line 1), and a list of files is read from the dataset of the Google cluster (line 2) to simulate the load requirements of a real cloud data center;

[0073] Within each time slot, the system environment is first initialized, including the configuration information of physical machines and virtual machines. In the `con_system` method, a file size list and a file access rate list are initialized first. Then, based on the file access rate and predefined resource utilization, the resource load for each file block is calculated. Specifically, CPU requirements are obtained by multiplying the file access rate by the predefined CPU utilization. IO requirements are multiplied by a random number based on the CPU requirements to simulate the volatility of IO resource usage. Network requirements are also multiplied by a random number based on the CPU requirements to simulate the volatility of network resource usage. For each file block and its copy, the calculated CPU, IO, and network resource requirements are added to `block_load_list` (line 4). During each iteration, a population is generated and initialized using round-robin and randomization methods to ensure the diversity of the solution set. During initialization, the physical bound load calculation method is used. The system traverses each virtual machine and, based on its assigned physical machine index, accumulates the virtual machine's resource requirements onto the corresponding physical machine, thus obtaining the total resource usage for each physical machine. To ensure the rationality of resource allocation and the stability of the system, the system implements resource constraint checks specifically as follows: if any resource ( If the usage of CPU, I / O, or network resources on a virtual machine or physical machine exceeds its configured capacity, the corresponding virtual machine placement method will be considered infeasible (line 7). Next, single-point crossover is performed on the virtual machines and block chromosomes, and different mutation and block replacement operations are executed to increase population diversity. Virtual machine growth and shrinking methods are used to adjust the number of virtual machines to explore different virtual machine configurations and seek better resource allocation schemes. The virtual machine growth method provides more resource allocation options by increasing the number of virtual machines, potentially reducing the load on individual virtual machines. The virtual machine shrinking method simplifies resource management by reducing the number of virtual machines, but requires reallocating data blocks from deleted virtual machines to other virtual machines to avoid resource overload. When performing a virtual machine shrinking operation, the system first releases the CPU, I / O, and network resources occupied by the deleted virtual machine. Subsequently, the system reallocates the data blocks from the deleted virtual machine to other virtual machines. This process affects the CPU, I / O, and network resource usage of the receiving virtual machine. The system must ensure that after reallocation, the new virtual machine will not affect overall performance due to resource overload, keeping resource usage within an acceptable range. After the mutation operation, the resource allocation of the solution changes.To ensure that the new solution still meets resource constraints and can accurately assess its fitness, the system recalculates the CPU, I / O, and network resource usage of each virtual machine and physical machine based on the current solution. This is done by traversing all file blocks, accumulating the resource requirements of the virtual machines allocated to each block, and then accumulating the resource usage of the physical machines allocated to the virtual machines (lines 8-9). Subsequently, the fitness of the offspring is evaluated and merged with the original population, and a resource constraint check is performed. The system traverses all physical machines and compares whether their CPU, I / O, and network resource usage exceeds the configured capacity. Then, it traverses all virtual machines and compares whether their CPU, I / O, and network resource usage exceeds the configured capacity. This ensures that each mutated solution is reasonable in terms of resource allocation and achieves the optimization goals in terms of overall performance and resource utilization (lines 10-11).

[0074] To guide the population toward the Pareto optimal front, pre-generated reference points are introduced. The current solution set is projected onto these reference points, and non-dominated sorting and crowding distance are calculated. The next generation of the population is then selected based on the correlation between these two points (lines 12-13). To ensure convergence and global search capability, the fitness of the initial population is saved during initialization, and the population quality is improved through multiple iterations. Next, the optimal solution is selected from the merged population as the next generation, and the current Pareto front is saved (lines 14-15). Finally, normalization ensures the balance between different optimization objectives, and the optimal virtual machine placement strategy is selected (lines 17-18).

[0075] The method for placing the virtual machine includes the following steps;

[0076] Step S1: First, deploy the virtual machine resource management module in the large-scale Hadoop cluster environment and configure the initial resource information of physical machines and virtual machines. The initial resource information includes CPU, memory, disk I / O and network bandwidth.

[0077] Step S2: The MO-VLD method uses an algorithm to monitor changes in resource demand and task load in the cluster in real time, and loads the corresponding file list or data task to ensure an accurate understanding of the actual situation of the cluster, laying the foundation for subsequent optimization.

[0078] In step S3, during resource management, the virtual machine placement method dynamically adjusts the allocation of virtual machines and the placement of data blocks based on the current task load and cluster resource usage. This intelligent adjustment maximizes resource utilization efficiency and reduces energy consumption under different loads.

[0079] Step S4: During the optimization process, the virtual machine placement method continuously analyzes the cluster's operating performance and performs policy feedback and adaptive optimization based on key indicators such as energy consumption, resource utilization, and data access latency to ensure that the cluster system is always in the best operating state.

[0080] This invention proposes a method for placing virtual machines in large-scale Hadoop clusters based on multi-objective optimization. It uses a novel multi-objective optimization method with Variable Length Double chromosome (MO-VLD) for placing virtual machines in large-scale Hadoop clusters. This method considers energy consumption, resource utilization, and file access latency as multiple optimization objectives. In particular, the system model adopted by the proposed method fully considers the dynamic changes in resource requirements, which can ensure that the Hadoop cluster can maintain good performance under high load.

[0081] This invention achieves efficient allocation of virtual machine resources in large-scale Hadoop clusters by combining variable-length dual-chromosome and multi-objective optimization algorithms. It significantly improves resource utilization and reduces overall energy consumption, exhibiting strong dynamic adaptability. It can automatically adjust allocation strategies based on changes in actual data load and resource requirements, ensuring stable and efficient cluster operation. The invention employs a non-dominated sorting genetic algorithm (NSGA-III) to achieve an efficient balance between resource utilization, energy consumption, and access latency. It possesses excellent global optimization capabilities and good flexibility and scalability, making it suitable for Hadoop clusters of various sizes, from small clusters to large cloud data centers, providing a stable virtual machine placement optimization solution. Main application scenarios include large-scale data processing environments, virtualization resource management in cloud computing platforms, and complex computing scenarios requiring dynamic resource scheduling, such as edge computing and cloud-edge collaborative computing. Attached Figure Description

[0082] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0083] Appendix Figure 1 This is a schematic diagram of the architecture of the present invention;

[0084] Appendix Figure 2 This is a schematic diagram illustrating the design principle of the MO-VLD method;

[0085] Appendix Figure 3 This is a diagram showing the convergence comparison of different methods in 40 physics machine scenarios;

[0086] Appendix Figure 4 This is a diagram showing the convergence comparison of different methods across 120 physics machine scenarios;

[0087] Appendix Figure 5 This is a diagram showing the convergence comparison of different methods across 200 physical machine scenarios;

[0088] Appendix Figure 6 These are schematic diagrams illustrating the Pareto solution set distributions for different methods;

[0089] Appendix Figure 7 This is a diagram showing the performance comparison of different methods under different numbers of physical machines;

[0090] Appendix Figure 8 This diagram illustrates the impact of different file counts on the performance of different methods. Detailed Implementation

[0091] like Figure 1 As shown, a multi-objective optimization-based method for placing virtual machines in a large-scale Hadoop cluster is described. This method is used to build a Hadoop cluster, whose system model includes multiple racks. Each rack contains multiple physical machines (PMs), and each PM runs multiple virtual machines (VMs). The racks are interconnected via a core switch, and the PMs within each rack are interconnected via switches. The Hadoop cluster uses switches for data block transmission and file replication, thereby achieving cross-rack data synchronization and load balancing. The optimization objective of the Hadoop cluster system model is to minimize the weighted sum of energy consumption, resource waste, and file access latency. Figure 2 As shown, the optimization objective employs a variable-length chromosome structure consisting of virtual machine chromosomes and block chromosomes. This chromosome structure is used to describe the location of virtual machine and data block replicas. The chromosome structure uses crossover and mutation operations to enhance the diversity of the solution space exploration for resource configuration during the virtual machine placement process in the Hadoop cluster.

[0092] In the system model of a Hadoop cluster, there are M physical machines in the Hadoop cluster, denoted as PM. i (i = 1, 2, ..., M), its resource allocation is denoted as a quadruple. These represent the CPU, memory, disk I / O, and network bandwidth of the PM, respectively; each PM i Running different numbers of VMs, which share a PM i Resources; per VM j The resource usage of (j = 1, 2, ..., N) is denoted as a quadruple. The resources allocated to a virtual machine are limited by the resource configuration of the physical machine, which is represented as follows: In the system model of the Hadoop cluster, the sum of the resources allocated to the virtual machines and the virtualization software overhead must not exceed the resource configuration of the physical machines to avoid excessive consumption of physical machine resources, as defined by the formula.

[0093] Where, ∑ pmi U pmi Indicates PM i The sum of resource consumption of all virtual machines; Indicates PM i Resource consumption of virtualization software.

[0094] In the system model of a Hadoop cluster, VM j Resource consumption cannot exceed the resources allocated to it, as defined by the formula.

[0095]

[0096] In the system model of a Hadoop cluster, the storage resource model is specifically as follows: Assume a file is split into multiple data blocks B. k (k = 1, 2, ..., K), so that its copies are stored on different virtual machines, i.e., the file is represented as a concatenation of blocks, defined by the formula...

[0097]

[0098] File size fSize fu With the size of the data block fbSize fu The number of data blocks is determined by the following definition:

[0099] To improve data availability and system fault tolerance, each data block is stored on multiple replicas on different virtual machines. k ; file f u Each data block There are multiple copies, which are defined as follows:

[0100]

[0101] Among them, fR fu Represents file f u The number of copies.

[0102] PM i Multiple copies of the same data block cannot be stored simultaneously to avoid wasting storage resources; this constraint is expressed as follows:

[0103]

[0104] In the system model of a Hadoop cluster, the energy consumption model is as follows:

[0105] Let PM i The energy consumption of PM is affected by CPU, network bandwidth, and disk I / O utilization. i Total energy consumption is defined by the formula

[0106]

[0107] in, and These represent the energy consumption of CPU, network bandwidth, and disk I / O, respectively.

[0108] Specifically, CPU energy consumption is defined by the formula as follows:

[0109]

[0110] Where, α i and β i They represent PM i Low CPU load and high CPU load level coefficients; Prh = P max -P idle λ represents the difference between the highest and lowest power of the physical machine. i Indicates PM i The load threshold is set, and when the load exceeds the threshold, the CPU will switch from the low load CPU power consumption model to the high load CPU power consumption model.

[0111] Network bandwidth energy consumption is defined as

[0112]

[0113] in, γ represents network bandwidth utilization. i This represents the impact coefficient of network bandwidth utilization on energy consumption; disk I / O energy consumption is defined as...

[0114]

[0115] in, This represents disk I / O utilization, δ i This represents the coefficient indicating the impact of disk I / O utilization on energy consumption.

[0116] In the system model of a Hadoop cluster, physical machines provide computing, storage, and network resources for multiple virtual machines. During the operation of a Hadoop cluster, resource waste caused by inefficient or unreasonable resource utilization is expressed by the relevant formulas in the resource utilization model.

[0117]

[0118] in, Indicates PM i Resource utilization, including CPU, disk bandwidth and network bandwidth utilization; ∈ is a positive coefficient used to avoid unreasonable extreme cases when the standard deviation is zero; Indicates PM i The standard deviation of the utilization rates of various resources is used to measure resource imbalance, and is defined by the formula:

[0119] in,

[0120] The average utilization rate of each resource is defined as follows:

[0121]

[0122] Therefore, PM i The sum of the utilization rates of all resources is defined as

[0123]

[0124] In a Hadoop cluster system model, file data transmission between physical machines requires multiple switches and physical links. The system model uses switch latency and physical link latency to evaluate file access latency between virtual machines. The specific file access latency model of the system model is as follows:

[0125] Let the switch delay T switch If the processing time required for a data block to pass through the switch is given, then the file access latency is the PM of the physical machine where the virtual machine resides. dn To the physical machine PM where the target virtual machine is located vm The sum of all switch delays and physical link delays between them, expressed in the formula table, is defined as follows:

[0126]

[0127] Where N represents the number of switches in the transmission path. This represents the delay of the k-th physical link; when there are I data blocks, the average transmission delay L of all data blocks is... avg Represented as

[0128]

[0129] The optimization objective of the Hadoop cluster system model is an NP-hard multi-objective optimization problem. When solving this multi-objective optimization problem, its formal definition is...

[0130] min(w1Z1+w2Z2+w3Z3) st(1),(2),(6) (17)

[0131] in,

[0132] w1, w2, and w3 represent the weights of the three optimization objectives: energy consumption, resource waste, and file access latency, respectively; Virtual Machine Chromosome C vm It is a two-dimensional array, defined as follows:

[0133]

[0134] In this table, the first row of each column represents the physical machine on which the virtual machine is placed, and the second row of each column represents the type of virtual machine, denoted as VMType. j ∈{1,2,3,4,5}.

[0135] Block chromosome C block This is a one-dimensional array representing the placement location of data block replicas, defined as follows:

[0136] C block ={VM1,VM2,...,VM n} (19)

[0137] Each element represents the virtual machine where the data block replica is placed;

[0138] During the initialization phase, C vm Virtual machines are placed on physical machines using a round-robin method, and the virtual machine type is randomly assigned; C block By generating data block replicas in a round-robin and random manner, the distribution of data block replicas is ensured to meet the requirements of the Hadoop cluster for the number of replicas. The process in the initialization phase guarantees the diversity in the solution space, so that the MO-VLD method for solving multi-objective optimization problems can fully explore different strategies during the runtime phase.

[0139] During evolution, chromosome structure utilizes a two-stage crossover and mutation process to generate new solutions and optimize the existing solution set; the two stages are:

[0140] Phase 1: Single-point crossover of virtual machine chromosomes; randomly selecting a cut point p, exchanging fragments of different parent virtual machine chromosomes to generate child virtual machine chromosomes; this process is expressed by the formula:

[0141] The second stage involves single-point crossover of the chromosome blocks; a random cutting point q is selected, and segments from different parent chromosome blocks are exchanged to generate daughter chromosome blocks; this process is expressed by the following formula:

[0142]

[0143] The mutation operation of the chromosome structure refers to increasing the diversity of solutions by randomly changing the genes in the chromosome;

[0144] In the chromosome structure, the mutation operations on virtual machines and block chromosomes are defined as follows: For virtual machine chromosomes, during the virtual machine mutation phase, the strategy to enhance population diversity and optimization capabilities is as follows: First, the expansion and contraction mutations of virtual machines adapt to dynamic resource demands by adjusting the number of virtual machines; whereby the number of virtual machines is initially set to... The probability of increasing or decreasing provides greater resource configuration flexibility for the solution; then, the virtual machine replacement mutation traverses the position of each element of the virtual machine chromosome and randomly modifies its value with a probability set to 0.5; on average, this mutation operation will cause half of the virtual machine allocations and template types to change; the method of traversing the virtual machine chromosome includes allocations and templates;

[0145] For block chromosomes, a block replacement operation is introduced: by changing the location of data block replicas, their storage location is optimized, thereby reducing data transfer latency. Specifically, when a data block replica is located in a virtual machine, the block replacement operation will reallocate the replica to another virtual machine with a probability set to 0.5.

[0146] When solving multi-objective optimization problems, the objective space is divided into multiple regions, each corresponding to a reference point. Reference points are associated and normalized. Each reference point is a predefined objective vector representing a potential solution under different objective weights. Specifically, during the evolution of each generation of the population, the current solution set is projected onto pre-generated reference points and associated based on the distance between them. The nearest reference point attracts solutions to move towards it, thus prioritizing solutions associated with sparse reference points. This reference-point-based selection operator maintains population diversity and ensures a uniform distribution of solutions, preventing solutions from concentrating in certain objective regions and neglecting optimization in other dimensions.

[0147] When solving multi-objective optimization problems, a normalization operation is introduced to compare different optimization objectives on the same scale. Specifically, this involves transforming the performance indicators of individuals in terms of energy consumption, resource utilization, and file access latency into the same order of magnitude, thereby eliminating the order-of-magnitude differences between different optimization objectives. This process is defined as follows:

[0148]

[0149] Where, r i r represents the weighted objective value of the i-th scheme; 1,i r 2,i and r 3,irepresents energy consumption, resource waste, and file access latency, respectively; n represents the total number of Pareto solutions.

[0150] The MO-VLD method achieves global optimization of multidimensional objectives by combining a reference point association mechanism with normalization operations.

[0151] The MO-VLD method includes the following steps;

[0152] Step 1: Chromosome Construction. During population initialization, a set of decision variables is generated for each individual (chromosome) according to a predetermined process. The number of virtual machines (vmNumber) is randomly determined based on system configuration, ensuring it is no less than the number of physical machines. Then, a double-loop and round-robin allocation strategy is used to generate data block allocation schemes, with random shuffling added at appropriate locations to guarantee solution diversity. Specifically, the `generateRoundRobin2Dshuffle` function first generates a continuous list of values ​​in a one-dimensional round-robin manner using `generateRoundRobin1D`. This list is then divided into multiple sub-segments according to a specified block size (i.e., replication factor). Each sub-segment is randomly shuffled, and all sub-segments are shuffled a second time, ultimately forming a two-dimensional allocation pattern. Subsequently, the `serialize2D` function serializes this two-dimensional list into a one-dimensional list for subsequent processing and storage.

[0153] Step 2: Generate corresponding sequences for virtual machines (vm) and virtual machine types (vmtype). Using the same round-robin strategy as in Step 1 (generateRoundRobin1D), assign a list of virtual machine IDs based on the number of physical machines and a list of virtual machine types based on the number of available types (4 in this case) to the vmNumber virtual machines. Then, randomly shuffle these two lists to obtain a dispersed and diverse virtual machine mapping and type allocation scheme in the initial state.

[0154] Step 3: After constructing and processing the block, vm, and vmtype, they are integrated and stored in a chromosome dictionary, representing the complete chromosome structure of the individual. The generated chromosome is then cleaned by calling the removeEmptyVms function (e.g., removing unused virtual machine entries) to ensure the validity of the chromosome structure. Finally, the chromosome is added to the solution list of the population. Simultaneously, multi-objective optimization information such as dominance relationships and crowding distances is initialized for each solution in the population. The corresponding algorithm is expressed in pseudocode as follows:

[0155]

[0156] The specific algorithm flow is as follows:

[0157] First, the initial maximum time slot T, maximum number of iterations X, and population size N are initialized (line 1), and a list of files is read from the dataset of the Google cluster (line 2) to simulate the load requirements of a real cloud data center;

[0158] Within each time slot, the system environment is first initialized, including the configuration information of physical machines and virtual machines. In the `con_system` method, a file size list and a file access rate list are initialized first. Then, based on the file access rate and predefined resource utilization, the resource load for each file block is calculated. Specifically, CPU requirements are obtained by multiplying the file access rate by the predefined CPU utilization. IO requirements are multiplied by a random number based on the CPU requirements to simulate the volatility of IO resource usage. Network requirements are also multiplied by a random number based on the CPU requirements to simulate the volatility of network resource usage. For each file block and its copy, the calculated CPU, IO, and network resource requirements are added to `block_load_list` (line 4). During each iteration, a population is generated and initialized using round-robin and randomization methods to ensure the diversity of the solution set. During initialization, the physical bound load calculation method is used. The system traverses each virtual machine and, based on its assigned physical machine index, accumulates the virtual machine's resource requirements onto the corresponding physical machine, thus obtaining the total resource usage for each physical machine. To ensure the rationality of resource allocation and the stability of the system, the system implements resource constraint checks specifically as follows: if any resource ( If the usage of CPU, I / O, or network resources on a virtual machine or physical machine exceeds its configured capacity, the corresponding virtual machine placement method will be considered infeasible (line 7). Next, single-point crossover is performed on the virtual machines and block chromosomes, and different mutation and block replacement operations are executed to increase population diversity. Virtual machine growth and shrinking methods are used to adjust the number of virtual machines to explore different virtual machine configurations and seek better resource allocation schemes. The virtual machine growth method provides more resource allocation options by increasing the number of virtual machines, potentially reducing the load on individual virtual machines. The virtual machine shrinking method simplifies resource management by reducing the number of virtual machines, but requires reallocating data blocks from deleted virtual machines to other virtual machines to avoid resource overload. When performing a virtual machine shrinking operation, the system first releases the CPU, I / O, and network resources occupied by the deleted virtual machine. Subsequently, the system reallocates the data blocks from the deleted virtual machine to other virtual machines. This process affects the CPU, I / O, and network resource usage of the receiving virtual machine. The system must ensure that after reallocation, the new virtual machine will not affect overall performance due to resource overload, keeping resource usage within an acceptable range. After the mutation operation, the resource allocation of the solution changes.To ensure that the new solution still meets resource constraints and can accurately assess its fitness, the system recalculates the CPU, I / O, and network resource usage of each virtual machine and physical machine based on the current solution. This is done by traversing all file blocks, accumulating the resource requirements of the virtual machines allocated to each block, and then accumulating the resource usage of the physical machines allocated to the virtual machines (lines 8-9). Subsequently, the fitness of the offspring is evaluated and merged with the original population, and a resource constraint check is performed. The system traverses all physical machines and compares whether their CPU, I / O, and network resource usage exceeds the configured capacity. Then, it traverses all virtual machines and compares whether their CPU, I / O, and network resource usage exceeds the configured capacity. This ensures that each mutated solution is reasonable in terms of resource allocation and achieves the optimization goals in terms of overall performance and resource utilization (lines 10-11).

[0159] To guide the population toward the Pareto optimal front, pre-generated reference points are introduced. The current solution set is projected onto these reference points, and non-dominated sorting and crowding distance are calculated. The next generation of the population is then selected based on the correlation between these two points (lines 12-13). To ensure convergence and global search capability, the fitness of the initial population is saved during initialization, and the population quality is improved through multiple iterations. Next, the optimal solution is selected from the merged population as the next generation, and the current Pareto front is saved (lines 14-15). Finally, normalization ensures the balance between different optimization objectives, and the optimal virtual machine placement strategy is selected (lines 17-18).

[0160] The method for placing the virtual machine includes the following steps;

[0161] Step S1: First, deploy the virtual machine resource management module in the large-scale Hadoop cluster environment and configure the initial resource information of physical machines and virtual machines. The initial resource information includes CPU, memory, disk I / O and network bandwidth.

[0162] Step S2: The MO-VLD method uses an algorithm to monitor changes in resource demand and task load in the cluster in real time, and loads the corresponding file list or data task to ensure an accurate understanding of the actual situation of the cluster, laying the foundation for subsequent optimization.

[0163] In step S3, during resource management, the virtual machine placement method dynamically adjusts the allocation of virtual machines and the placement of data blocks based on the current task load and cluster resource usage. This intelligent adjustment maximizes resource utilization efficiency and reduces energy consumption under different loads.

[0164] Step S4: During the optimization process, the virtual machine placement method continuously analyzes the cluster's operating performance and performs policy feedback and adaptive optimization based on key indicators such as energy consumption, resource utilization, and data access latency to ensure that the cluster system is always in the best operating state.

[0165] Example:

[0166] The algorithm used in this example was implemented on a workstation equipped with an Intel(R) Xeon(R) Silver 4208 CPU and an NVIDIA GeForce GTX 3090 GPU, with a CPU frequency of 2.1 GHz and 32 GB of memory. The experiment was conducted using Python 3.9, building a large-scale Hadoop cluster environment configured to resemble a typical cloud data center, and utilizing the Google Cluster Dataset. This dataset records runtime data acquired from Borg cells over a 7-hour period, reflecting task scheduling and resource utilization in a cloud data center. The environment was initialized based on task arrival times within each time slot to simulate scenarios with dynamically changing loads. Specifically, the file system settings are shown in Table 1, and the physical machine and virtual machine settings are shown in Tables 2 and 3, respectively. Furthermore, this invention compares the proposed MO-VLD method with the following five methods.

[0167] (1) Second generation Non-dominated Sorting Genetic Algorithm (NSGA-II): Select offspring by non-dominated sorting and crowding.

[0168] (2) Multi-Objective Particle Swarm Optimization (MOPSO): Utilizes the global search capability of particle swarms to optimize multiple objectives.

[0169] (3) Multi-Objective Simulated Annealing (MOSA): Finds an approximate optimal solution through a temperature-controlled stepwise annealing process.

[0170] (4) Best-Fit Decreasing (BFD): Prioritize placing virtual machines on physical machines that best meet their resource requirements.

[0171] (5) First-Fit Decreasing (FFD): The virtual machine is placed in the first physical machine that can meet its resource requirements according to the order priority.

[0172] In the experiments, the population size for all methods was set to 200, with 300 iterations, and a round-robin and randomization strategy was used for population initialization. The MO-VLD and NSGA-II methods employed a two-stage crossover strategy. The MO-VLD method implemented mutation operations through virtual machine expansion, reduction, allocation, and block replacement, while the NSGA-II method used only block replacement for mutation. The mutation probability was 0.25 for both methods. The inertia weight, cognitive weight, and social weight for the MOPSO method were set to 0.5, 1.5, and 1.5, respectively. The initial temperature and cooling rate for the MOSA method were 100 and 0.99, respectively.

[0173] Table 1 File System Settings

[0174]

[0175] Table 2 Physical Machine Settings

[0176]

[0177] Table 3 Virtual Machine Settings

[0178]

[0179] First, this example compares the convergence and Pareto solution set distribution of different methods. The number of files is set to 200, and the number of physical machines to be 40, 120, and 200, respectively, to compare the convergence and solution quality of different methods. Figure 3 , Figure 4 and Figure 5 The convergence of different methods was compared in scenarios with 40, 120, and 200 physics machines. All methods gradually converged with increasing iteration count. The MO-VLD method achieved fast convergence in fewer iterations, and its final optimization performance was superior to other methods. This is because the MO-VLD method employs a dual-chromosome structure and a two-stage crossover and mutation operation, which increases the diversity of solutions and avoids the method getting trapped in local optima. Simultaneously, the MO-VLD method introduces a reference point association mechanism, which better balances multiple optimization objectives, ensuring that each objective is effectively optimized. In contrast, while the NSGA-II method has a faster convergence speed in the early stages of iteration, its overly simple crossover and mutation operation design leads to slightly inferior optimization performance in later stages. The MOPSO method is prone to getting trapped in local optima, while the MOSA method, although it easily obtains good solutions in early iterations, experiences slow convergence due to its annealing process. Specifically, as... Figure 3As shown, when the number of physical machines is 40, compared with the NSGA-II, MOPSO, and MOSA methods, the proposed MO-VLD method reduces energy consumption by approximately 3.12%, 7.9%, and 4.3%, respectively; improves resource utilization by approximately 1.4%, 7.6%, and 9%, respectively; and reduces file access latency by approximately 0.7%, 17.2%, and 22.2%, respectively. Figure 4 As shown, when the number of physical machines is 120, compared with the NSGA-II, MOPSO, and MOSA methods, the proposed MO-VLD method reduces energy consumption by approximately 4.47%, 6.70%, and 9.3%, respectively, and improves resource utilization by [missing information].

[0180] These figures represent increases of approximately 5.14%, 5.25%, and 13.04%, respectively, while file access latency decreased by approximately 1.37%, 4.17%, and 6.91%. Figure 5 As shown, when the number of physical machines is 200, compared with the NSGA-II, MOPSO and MOSA methods, the proposed MO-VLD method reduces energy consumption by approximately 1.0%, 1.8% and 5.8%, respectively; improves resource utilization by approximately 0.9%, 5.9% and 7.4%, respectively; and reduces file access latency by approximately 1.3%, 8.5% and 10.3%, respectively.

[0181] Figure 6 The Pareto solution set distributions of different methods were compared. Among these methods, the performance of MOSA, MOPSO, NSGA-II, and the proposed MO-VLD method improved in that order. Compared to the other three methods, the Pareto solution set of the MO-VLD method is closer to the origin of the 3D coordinate axes, indicating that the MO-VLD method obtains higher quality solutions. This is because the MO-VLD method introduces crossover and mutation operations, thus its global search capability is superior to other methods. The number of Pareto solutions also increases with the number of physics machines, indicating that the number of potentially feasible solutions increases with the complexity of the scene.

[0182] Furthermore, this example compares the performance of different methods under different numbers of physical machines. The number of physical machines is set to 40, 80, 120, 160, and 200, and the number of virtual machines is twice the number of physical machines. When the number of physical machines is less than 100, the number of racks is 5; otherwise, the number of racks is 8. These settings simulate the configuration of a large-scale cluster. This invention normalizes and weights three performance indicators: energy consumption, resource waste, and file access latency, and then compares the performance of different methods. Figure 7As shown, the proposed MO-VLD method has the lowest weighted objective value. This is because the MO-VLD method employs a dual-chromosome structure, allowing for synchronous optimization of virtual machine and data block replica placement, significantly reducing cross-rack data transfer and uneven use of physical machine resources. Simultaneously, the two-stage crossover and mutation operations in the MO-VLD method enhance its ability to explore the solution space. Therefore, the MO-VLD method performs excellently in multi-objective optimization problems, maintaining good and stable optimization results regardless of the number of physical machines. The weighted objective value of the NSGA-II method is only about half that of the FFD and BFD methods, and about 0.05 lower than the MOPSO and MOSA methods. Although the NSGA-II method also performs relatively stably in various scenarios, its relatively simple crossover and mutation operations make it difficult to effectively balance multiple optimization objectives in large-scale cluster scenarios, resulting in slightly inferior performance compared to the MO-VLD method, especially when the number of physical machines increases to 200. The performance of the MOPSO and MOSA methods consistently lags behind the proposed MO-VLD method. This is because the MOPSO method is prone to getting trapped in local optima during particle swarm search, making it difficult to achieve global equilibrium in multi-objective optimization problems. While the MOSA method possesses some global search capability, its annealing process results in excessive runtime overhead, leading to poor performance with large-scale clusters. FFD and BFD, being heuristic algorithms, severely lack global optimization capabilities when facing large-scale clusters, thus exhibiting the worst optimization results.

[0183] Finally, this example evaluates the impact of different file counts on the performance of different methods. The number of physical machines was set to 200, and the number of files was set to 40, 80, 120, 160, and 200. Figure 8 As shown, the proposed MO-VLD method achieves optimal performance across different file count scenarios. In terms of energy consumption, MO-VLD is lower than other methods in all file count scenarios, and it maintains low energy consumption even as the number of files increases. This is because the dual-chromosome structure in the MO-VLD method enables…

[0184] The optimal placement of virtual machines and data blocks avoids resource waste or overuse. Compared to NSGA-II, MOPSO, MOSA, FFD, and BFD methods, the MO-VLD method achieves average performance improvements of approximately 0.4%, 5.0%, 5.6%, 23.6%, and 26.2% in different scenarios. As the number of files increases, the resource waste of all methods decreases. Compared to other methods, the MO-VLD method maintains consistently low resource waste across different file counts. This is because the MO-VLD method employs dynamic resource scheduling and block replacement mechanisms, allowing for timely adjustments when resource demands change. Compared to NSGA-II, MOPSO, MOSA, FFD, and BFD methods, the MO-VLD method achieves average performance improvements of approximately 16.2%, 26.0%, 28.9%, 35.1%, and 38.4% in different scenarios. Regarding file access latency, although the performance differences between methods are not significant, the MO-VLD method still demonstrates good performance. This demonstrates that the MO-VLD method can effectively reduce data transfer time and improve file access efficiency in the cluster. Specifically, compared with the NSGA-II, MOPSO, MOSA, FFD, and BFD methods, the MO-VLD method achieves an average performance improvement of approximately 1.8%, 8.9%, 8.5%, 7.5%, and 6.2% in different scenarios.

Claims

1. A method for large-scale Hadoop cluster virtual machine placement based on multi-objective optimization, characterized in that: The placement method is used for assembling a Hadoop cluster, a system model of the Hadoop cluster comprises a plurality of racks, a plurality of physical machines (PMs) are arranged in each rack, and each physical machine runs a plurality of virtual machines (VMs); the racks are interconnected through a core switch, and the PMs in the rack are interconnected through a switch; the Hadoop cluster transmits data blocks and replicates file copies through the switch, so as to realize data synchronization and load balancing across the racks; an optimization target of the system model of the Hadoop cluster is to minimize the weighted sum of energy consumption, resource waste and file access delay of the Hadoop cluster, and a method used for the optimization target uses a variable-length chromosome structure composed of a virtual machine chromosome and a block chromosome, the chromosome structure is used for expressing the positions of the placed virtual machines and data block copies, and the chromosome structure uses crossover and mutation operations to enhance the diversity of the solution space exploration of resource configuration in the virtual machine placement process of the Hadoop cluster; The optimization target of the system model of the Hadoop cluster is an NP-hard multi-objective optimization problem, the sum of the allocated resources of the virtual machine and the virtualization software overhead cannot exceed the resource configuration of the physical machine, the resource consumption of the virtual machine cannot exceed the allocated resources, and the physical machine cannot simultaneously store multiple copies of the same data block; The formal definition of the target optimization problem is min (w1Z1+w2Z2+w3Z3) (17) wherein w1, w2 and w3 represent the weights of the three optimization targets of energy consumption, resource waste and file access delay, respectively; Virtual machine chromosome C vm is a two-dimensional array defined as Wherein, the first row of each column represents the physical machine in which the virtual machine is placed, and the second row of each column represents the type of the virtual machine, denoted as VMTypej∈{1,2,3,4,5}; Block Chromosome C block is a one-dimensional array, representing the placement positions of data block copies, which is defined as C block = {VM1, VM2,..., VM n} (19) Wherein, each element represents the virtual machine in which the data block copy is placed; In the initialization phase, C vm The virtual machines are placed to the physical machines by rotation and the virtual machine types are randomly assigned. block The data blocks are generated by rotation and randomness, ensuring that the distribution of the data block copies meets the requirements of the number of copies of the Hadoop cluster, and the diversity in the solution space is ensured by the process in the initialization phase, so that the MO-VLD method for solving multi-objective optimization problems can fully explore different strategies in the running phase. In the evolution process of the chromosome structure, two-stage crossover and mutation operations are used to generate new solutions and optimize the existing solution set; the two stages are as follows: The first stage: single-point crossover is performed on the virtual machine chromosome; a cutting point p is randomly selected, and the fragments of different parent virtual machine chromosomes are exchanged to generate offspring virtual machine chromosomes; The second stage: single-point crossover is performed on the block chromosome; a cutting point q is randomly selected, and the fragments of different parent block chromosomes are exchanged to generate offspring block chromosomes; The MO-VLD method comprises the following steps; Step one, construction of the chromosome, when initializing the entire population, for each individual of the cluster, that is, the chromosome, generate the combination of its decision variables according to the established process; the number of virtual machines vmNumber is randomly determined according to the system configuration, and it is ensured that it is not less than the number of physical machines, then a double loop and a rotation allocation strategy are used to generate an allocation scheme of the data block block, and a random disturbance process is added at a proper position, so as to ensure the diversity of the solution; Step two, generate the corresponding sequence for virtual machine vm and virtual machine type vmtype, adopt the same rotation strategy as step one, vmNumber virtual machine allocation based on the virtual machine ID list of the physical machine number range and the virtual machine type list based on the available type number, randomly shuffle the two lists respectively to get the initial state of the dispersed and diverse virtual machine mapping and type allocation scheme; Step three, after completing the construction and processing of block, vm, vmtype, integrate them into a chromosome dictionary, which represents the complete chromosome structure of the individual, then clean up the generated chromosome, remove unused virtual machine entries, and ensure the effectiveness of the chromosome structure; finally, add the chromosome to the solution list of the population, and initialize the dominance relationship and crowded distance of each solution in the population.

2. The method of claim 1, wherein: In the system model of Hadoop cluster, there are M physical machines, denoted as PM i (i = 1, 2,... M), whose resource configuration is denoted as a four-tuple CPU, memory, disk I / O and network bandwidth of the PM, respectively; each PM i runs different number of VMs, which share the resources of the PM i ; each VM j (j = 1, 2,... N) has a resource usage denoted as a four-tuple The resource allocated to a VM is limited by the resource configuration of the PM, which is denoted as In the system model of the Hadoop cluster, the sum of the allocated resources of the virtual machine and the virtualization software overhead cannot exceed the resource configuration of the physical machine, so as to avoid excessive consumption of the resources of the physical machine, and the formula is defined as wherein∑ pmi U pmi represents the PM i the sum of resource consumption of all virtual machines on the server; represents the PM i resource consumption of the virtualization software on the server; In the system model of Hadoop cluster, the resource consumption of VM j cannot exceed the allocated resources, which is defined as 3. The method of claim 2, wherein: In the system model of Hadoop cluster, the storage resource model is specifically: suppose a file is cut into multiple data blocks B k (k = 1, 2, …, K), so that its copy is stored on different virtual machines, that is, the file is represented as the connection of blocks, which is defined in the formula as Size of file fSize fu Size of data block fbSize fu Number of data blocks is determined, which is defined as To improve data availability and system fault tolerance, each data block is stored on multiple replicas on different virtual machines. k ; file f u Each data block There are multiple copies, which are defined as follows: where fR fu represents the number of copies of file f u . PM i The fact that multiple copies of the same data block cannot be stored simultaneously to avoid wasting storage resources is denoted as 4. The method of claim 3, wherein: In the system model of the Hadoop cluster, the energy consumption model is specifically: Let PM i be the energy consumption of PM i , which is affected by CPU, network bandwidth and disk I / O utilization. Then the total energy consumption of PM i is defined as wherein, and represent the energy consumption of CPU, network bandwidth and disk I / O, respectively; Specifically, the CPU energy consumption is defined in the formula Where, α i and β i They represent PM i Low CPU load and high CPU load level coefficients; Prh = P max -P idle λ represents the difference between the highest and lowest power of the physical machine. i Indicates PM i The load threshold is set, and when the load exceeds the threshold, the CPU will switch from the low load CPU power consumption model to the high load CPU power consumption model. The network bandwidth energy consumption is defined as wherein, denotes the network bandwidth utilization rate, γ i denotes the influence coefficient of the network bandwidth utilization rate on energy consumption; The disk I / O energy consumption is defined as wherein, represents the disk I / O utilization rate, δ i represents the impact coefficient of the disk I / O utilization rate on energy consumption.

5. The method of claim 4, wherein: In the system model of the Hadoop cluster, the physical machine provides computing, storage and network resources for multiple virtual machines; during the operation of the Hadoop cluster, inefficient or unreasonable resource utilization causes resource waste, which is expressed in the resource waste related formula in the resource utilization model where, represents the resource utilization of PM i , including CPU, disk bandwidth and network bandwidth utilization; ∈ is a positive coefficient to avoid unreasonable extreme cases when the standard deviation is zero; represents the standard deviation of each resource utilization of PM i , to measure the imbalance of resources, defined by the formula wherein, denotes the mean value of the resource utilization, which is defined as Thus, PM i The sum of the utilization of all resources is defined as 6. The method of claim 5, wherein: In the system model of the Hadoop cluster, the transmission of file data between physical machines needs to go through multiple switches and physical links, and the system model uses switch delay and physical link delay to evaluate the file access delay between virtual machines, then the file access delay model of the system model is specifically: Let the switch delay T switch If the processing time required for a data block to pass through the switch is given, then the file access latency is the PM of the physical machine where the virtual machine resides. dn To the physical machine PM where the target virtual machine is located vm The sum of all switch delays and physical link delays between them, expressed in the formula table, is defined as follows: wherein N represents the number of switches in the transmission path, denotes the delay of the kth physical link; When there are I data blocks, the average transmission delay L of all data blocks avg is represented as 7. The method of claim 1, wherein: When solving the multi-objective optimization problem, the target space is divided into multiple regions, each region corresponds to a reference point, the reference point is associated and normalized, and the reference point is a pre-defined target vector representing potential solutions under different target weights; Specifically: in the evolution process of each generation of the population, the current solution set will be projected onto the pre-generated reference points, and then associated according to the distance between them, then the solutions associated with sparse reference points are preferentially selected; the reference point-based selection operator is used to maintain the diversity of the population and ensure uniform distribution of solutions, thereby avoiding the concentration of solutions in certain target areas while ignoring optimization in other dimensions; when solving the multi-objective optimization problem, in order to compare different optimization targets on the same scale, a normalization operation is introduced, which is specifically: converting the performance indicators of individuals in energy consumption, resource utilization and file access delay to the same order of magnitude, thereby eliminating the magnitude difference between different optimization targets; defined as wherein r i represents the weighted objective value of the i-th scheme; r 1,i , r 2,i and r 3,i represent energy consumption, resource waste and file access delay, respectively; n represents the total number of the Pareto solution set; The MO-VLD method realizes global optimization of multi-dimensional targets by combining the reference point association mechanism and normalization operation.

8. The method of claim 1, wherein: The use method of the virtual machine placement method includes the following steps: Step S1, first deploy a virtual machine resource management module in a large-scale Hadoop cluster environment, configure the initial resource information of the physical machine and the virtual machine, and the initial resource information includes CPU, memory, disk I / O and network bandwidth; Step S2, the MO-VLD method uses an algorithm to monitor the changes in resource demand and task load in the cluster in real time and loads the corresponding file list or data task, ensuring accurate understanding of the actual situation of the cluster and laying a foundation for subsequent optimization; Step S3, in resource management, the virtual machine placement method dynamically adjusts the allocation of virtual machines and the placement position of data blocks according to the current task load and cluster resource usage; through intelligent adjustment, the cluster maximizes resource utilization efficiency and reduces energy consumption under different loads; Step S4, during the optimization process, the virtual machine placement method continuously analyzes the running effect of the cluster and performs policy feedback and adaptive optimization according to the key indicators of energy consumption, resource utilization rate and data access delay, ensuring that the cluster system is always in the best running state.

Images