Cluster resource scheduling method and device, equipment and storage medium

Abstract

The invention discloses a cluster resource scheduling method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring information of a plurality of to-be-allocated resource units and nodes in a cluster, randomly generating an initial allocation scheme of the to-be-allocated resource units and the nodes, and generating a first initial code string representing the initial allocation scheme; performing correction and fitness calculation processing on the first initial coding string; according to the first corrected code string, generating a new initial code string through a genetic algorithm cycle, and performing correction and fitness calculation processing on the new initial code string until a termination condition of the genetic algorithm is satisfied; and screening out an optimal corrected code string according to the individual fitness of all the generated corrected code strings, and taking an allocation scheme represented by the optimal corrected code string as a cluster resource allocation scheme so as to perform cluster resource scheduling according to the cluster resource allocation scheme. More reasonable cluster resource allocation is realized, and the cluster resource utilization rate is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cluster resource scheduling method, device, equipment and storage medium. Background Art

[0002] The rapid development of Internet technology has given rise to a new computing model, cloud computing, which allows computing and storage capabilities to be flexibly allocated according to actual needs. At present, containerization technology is usually used for resource isolation, and container orchestration tools are used for resource scheduling. In related technologies, scheduling schemes are selected based on resource scheduling requests and the resource utilization of a single node, but only the service quality of a single node is focused on, and the overall resource utilization of the cluster is not high. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a cluster resource scheduling method, device, equipment and storage medium, which can effectively improve the cluster resource utilization. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a cluster resource scheduling method, comprising:

[0005] Acquire information of a plurality of resource units to be allocated and nodes in a cluster, randomly generate an initial allocation scheme of the resource units to be allocated and the nodes, and generate a first initial code string representing the initial allocation scheme;

[0006] Performing correction and fitness calculation processing on the first initial code string; the correction and fitness calculation processing includes: correcting the first initial code string using a pre-constructed verification dictionary to obtain a first corrected code string, and determining the individual fitness of the first corrected code string;

[0007] Generate a new initial code string through a genetic algorithm loop according to the first corrected code string, and perform the correction and fitness calculation process on the new initial code string until the termination condition of the genetic algorithm is met;

[0008] The optimal corrected code string is screened out according to the individual fitness of all generated corrected code strings, and the allocation scheme represented by the optimal corrected code string is used as the cluster resource allocation scheme, so as to perform cluster resource scheduling according to the cluster resource allocation scheme.

[0009] Optionally, the verification dictionary is generated according to the deployment type of each node relative to each resource unit to be allocated; the deployment type includes allowed deployment and prohibited deployment;

[0010] The deployment type is determined according to the resource requirements of each of the resource units to be allocated and the idle resources of each node in the cluster.

[0011] Optionally, the correcting the first initial code string using a pre-built verification dictionary to obtain a first corrected code string includes:

[0012] Using the verification dictionary to verify the allocation of the resource units to be allocated in the first initial code on the nodes;

[0013] Target resource units to be allocated that do not conform to the verification dictionary are screened out, nodes are reallocated for the target resource units to be allocated according to the verification dictionary, and a code string is regenerated to obtain the first corrected code string.

[0014] Optionally, the step of cyclically generating a new initial code string through a genetic algorithm according to the first corrected code string includes:

[0015] Based on the first corrected code string and the fitness of each chromosome in the first corrected code string, a second initial code string is generated by using a genetic algorithm; the chromosome in the first corrected code string represents the allocation relationship between each resource unit to be allocated and each corresponding node;

[0016] The correction and fitness calculation process is performed on the second initial code string.

[0017] Optionally, the process of determining the individual fitness includes:

[0018] Determine the resource usage ratio of different types of resources of the resource unit to be allocated on the allocated node in the current corrected code string;

[0019] Sort the usage proportions of each resource corresponding to the resource unit to be allocated according to the numerical values, and configure weights for each type of resource; wherein, the higher the numerical value of the resource usage proportion, the higher the weight corresponding to the type of resource, and the lower the numerical value of the resource usage proportion, the lower the weight corresponding to the type of resource;

[0020] According to the resource usage ratio of the resource unit to be allocated in each type of resource and the weight corresponding to each type of resource, the average resource utilization rate of the resource unit to be allocated in the allocated node is obtained by weighted average calculation;

[0021] According to the average resource utilization rate of each resource unit to be allocated in the current corrected code string at the respective allocated node, determining the average resource utilization rate of the allocation scheme corresponding to the current corrected code string;

[0022] Determine the standard deviation of the current corrected code string according to the average resource utilization rate corresponding to all generated corrected code strings;

[0023] The standard deviation of the current corrected code string is used as the individual fitness of the current corrected code string.

[0024] Optionally, the resource usage ratio includes processor utilization, memory occupancy, disk occupancy ratio, and network bandwidth utilization.

[0025] Optionally, the step of selecting the optimal corrected coding string according to the individual fitness of all generated corrected coding strings includes:

[0026] Sort the standard deviations corresponding to all the corrected code strings generated;

[0027] The corrected code string with the smallest standard deviation is selected as the optimal corrected code string.

[0028] In a second aspect, the present application discloses a cluster resource scheduling device, comprising:

[0029] A first initial code string generation module, used to obtain information of a plurality of resource units to be allocated and nodes in a cluster, randomly generate an initial allocation scheme of the resource units to be allocated and the nodes, and generate a first initial code string representing the initial allocation scheme;

[0030] A correction module is used to perform correction and fitness calculation processing on the first initial code string; the correction and fitness calculation processing includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string;

[0031] A new initial code string generation module, used to generate a new initial code string through a genetic algorithm loop according to the first corrected code string, and perform the correction and fitness calculation process on the new initial code string until the termination condition of the genetic algorithm is met;

[0032] The screening module is used to screen out the optimal corrected code string according to the individual fitness of all generated corrected code strings, and use the allocation scheme represented by the optimal corrected code string as the cluster resource allocation scheme to perform cluster resource scheduling according to the cluster resource allocation scheme.

[0033] In a third aspect, the present application discloses an electronic device, including:

[0034] Memory, used to store computer programs;

[0035] The processor is used to execute the computer program to implement the aforementioned cluster resource scheduling method.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned cluster resource scheduling method when executed by a processor.

[0037] In the present application, information of multiple resource units to be allocated and nodes in a cluster is obtained, an initial allocation scheme of the resource units to be allocated and the nodes is randomly generated, and a first initial code string representing the initial allocation scheme is generated; correction and fitness calculation processing are performed on the first initial code string; the correction and fitness calculation processing includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string; according to the first corrected code string, a new initial code string is cyclically generated through a genetic algorithm, and the correction and fitness calculation processing is performed on the new initial code string until the termination condition of the genetic algorithm is met; the optimal corrected code string is screened out according to the individual fitness of all the generated corrected code strings, and the allocation scheme represented by the optimal corrected code string is used as the cluster resource allocation scheme, so as to perform cluster resource scheduling according to the cluster resource allocation scheme. It can be seen that by generating allocation plans for multiple resource units to be allocated and encoding the allocation plans into code strings, the genetic algorithm is used to simulate natural selection and genetic mechanisms to optimize resource allocation, consider the load balancing of the entire cluster, and improve the resource utilization of the entire cluster; and by correcting the initial code string through the verification dictionary, the invalid initial code string is corrected, which improves the speed of screening the optimal resource allocation plan, improves the efficiency of cluster resource scheduling, achieves more reasonable cluster resource allocation, and effectively improves cluster resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] FIG1 is a flow chart of a cluster resource scheduling method provided by the present application;

[0040] Figure 2 A specific diagram of a verification dictionary provided for this application;

[0041] Figure 3 A specific genetic algorithm flow chart provided for this application;

[0042] FIG4 is a flow chart of a specific cluster resource scheduling method provided by the present application;

[0043] FIG5 is a schematic diagram of the structure of a cluster resource scheduling device provided by the present application;

[0044] FIG6 is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In the related art, a scheduling scheme is selected according to a resource scheduling request and the resource utilization of a single node, but only the service quality of a single node is focused on, and the overall resource utilization of the cluster is not high. To overcome the above technical problems, the present application proposes a cluster resource scheduling method, which can more reasonably allocate cluster resources and effectively improve cluster resource utilization.

[0047] The present application embodiment discloses a cluster resource scheduling method. Referring to FIG1 , the method may include the following steps:

[0048] Step S11: obtaining information of a plurality of resource units to be allocated and nodes in a cluster, randomly generating an initial allocation scheme of the resource units to be allocated and the nodes, and generating a first initial code string representing the initial allocation scheme.

[0049] In this embodiment, the information of the resource units to be allocated and the nodes in the cluster is first determined, and an initial allocation scheme for the resource units to be allocated and the nodes is randomly generated, for example, the i-th resource unit to be allocated is allocated on the j-th node, and the initial allocation scheme is encoded to obtain a first initial code string. In some embodiments, the use of a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string may include: using the verification dictionary to verify the allocation of the resource units to be allocated in the first initial code on the nodes; screening out the target resource units to be allocated that do not conform to the verification dictionary, reallocating nodes for the target resource units to be allocated according to the verification dictionary, and regenerating the code string to obtain the first corrected code string.

[0050] It can be understood that, in order to use the genetic algorithm later, in this embodiment, the population is first initialized, that is, the relationship between the node and the resource unit is represented in a binary coding manner. Specifically, the relationship between the resource unit and the node is encoded, and the encoded individual is called a chromosome, which is the input of the genetic algorithm. The chromosomes corresponding to multiple resource units to be allocated constitute a coding string. For example, the chromosome is encoded in binary mode. For the chromosome set {100,010,001} containing three chromosomes 100, 010, and 001, the set indicates that the first resource unit is allocated to the first node, the second resource unit is allocated to the second node, and the third resource unit is allocated to the third node.

[0051] In some embodiments, the above-mentioned resource units to be allocated include resource units waiting for the first deployment and resource units that have been deployed. That is, the present application realizes the first deployment of resource units waiting for the first deployment by obtaining the information of resource units waiting for the first deployment and the information of nodes in the cluster. At the same time, by obtaining the information of resource units that have been deployed and the information of nodes in the cluster, it can support the redeployment of resource units that have been deployed, thereby realizing dynamic deployment of resource units, and can adjust the deployment strategy according to the situation of the resource units, so that resource scheduling is more in line with the resource demand situation of the resource units, thereby improving the performance of applications in the resource units.

[0052] Step S12: performing correction and fitness calculation processing on the first initial code string; the correction and fitness calculation processing includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string.

[0053] The above-mentioned verification dictionary is generated according to the deployment type of each node relative to each resource unit to be allocated; the deployment type includes allowed deployment and prohibited deployment; the above-mentioned deployment type is determined according to the resource requirements of each resource unit to be allocated and the idle resources of each node in the cluster. It can be understood that the allocation scheme corresponding to the randomly generated code string may allocate nodes to resource units that do not meet the hard conditions of resource units. For example, it is meaningless to allocate a resource unit that requires 4TB of disk space to a node that only has 2TB. It is meaningless to evaluate the fitness of these nodes that do not meet the resource unit; therefore, dictionary verification is used to verify and repair the allocation that does not meet the cluster configuration requirements, and the chromosomes corresponding to the nodes that do not meet the resource unit requirements are verified and corrected, so that the space resources of the nodes in the allocation scheme corresponding to the corrected code string at least meet the resource requirements of the resource unit.

[0054] For example Figure 2The figure shows a specific dictionary verification diagram, in which the verification dictionary corresponds to the node according to the resource requirements of the resource unit. For example, resource unit 1 and resource unit 2 are allowed to be scheduled to node 1, but resource unit 5 is not allowed to be scheduled to node 2. For the condition of 7 scheduling tasks and 3 nodes, the initialization code is {100,010,001,100,010,100,100}. Chromosomes 100, 010, and 001 remain unchanged after dictionary verification, because resource unit 1 is allowed to be allocated on the first node, resource unit 2 is also allowed to be allocated on the second node, and resource unit 3 is not specified to be allocated on node 3, which can be allowed by default; resource unit 4 is not allowed to be allocated on the first node, but resource unit 4 is allowed to be scheduled on node 2, so the fourth chromosome 100 is corrected to 010; through sequential verification, the initial code string {100,010,001,100,010,100,100} is verified by the verification dictionary as {100,010,001,010,001,100,100}, and the chromosomes after verification are more accurate for the scheduling of the genetic algorithm.

[0055] Step S13: Generate a new initial code string through a genetic algorithm loop according to the first corrected code string, and perform the correction and fitness calculation process on the new initial code string until the termination condition of the genetic algorithm is met.

[0056] After the initial code string is randomly generated for the first time, a new initial code string is subsequently generated using a genetic algorithm to predict the optimal allocation scheme, and the generated new initial code is corrected using a verification dictionary, and the individual fitness is calculated after correction; at the same time, after each new initial code is generated, it is determined whether the genetic algorithm termination condition is met, which can be a preset number of iterations. If the genetic algorithm termination condition is met, the operation is stopped, and if not, a new initial code string continues to be generated.

[0057] In some embodiments, the looping generation of a new initial code string through a genetic algorithm based on the first corrected code string may include: generating a second initial code string using a genetic algorithm based on the first corrected code string and the fitness of each chromosome in the first corrected code string; the chromosomes in the first corrected code string represent the allocation relationship between each resource unit to be allocated and each corresponding node; the second initial code string performs the correction and fitness calculation processing. That is, by retaining chromosomes with high fitness, a new initial code string can be generated based on the retained chromosomes to improve the performance of the newly generated initial code string with good individual fitness.

[0058] In some embodiments, the process of determining the individual fitness may include the following steps:

[0059] S21: Determine the resource usage ratio of different types of resources of the resource unit to be allocated on the allocated node in the current corrected code string; the resource usage ratio includes processor utilization, memory occupancy, disk occupancy ratio, and network bandwidth utilization.

[0060] That is, multiple resource indicators such as CPU, memory resources, disk and network bandwidth are used to analyze resource utilization, and cluster resource scheduling is better optimized by expanding resource indicators. By considering multiple resource indicators such as CPU, memory, network bandwidth and disk IO, and assigning weights to these indicators, more balanced resource allocation can be achieved, reducing the overload of a single node in the cluster, and achieving comprehensive resource scheduling.

[0061] S22: Sort the usage proportions of each type of resource corresponding to the resource units to be allocated according to their numerical values, and configure weights for each type of resource; wherein, the higher the numerical value of the resource usage proportion, the higher the weight corresponding to this type of resource, and the lower the numerical value of the resource usage proportion, the lower the weight corresponding to this type of resource.

[0062] It is understandable that the chromosome after verification can meet the resource scheduling plan. In the genetic algorithm, it is necessary to calculate the fitness to evaluate the performance of each node in solving specific problems. Here, in order to make the fitness evaluation more accurate, cpu, memory resources, disk and network bandwidth are used as indicators, and different weights are assigned to the indicators to represent the degree of dependence of resource units on different resources of the node. That is, the resource utilization of each indicator of the node is calculated, and different weights are assigned according to the size of each indicator. The weight represents the dependence of the resource unit on the resource indicator of the node. Therefore, when analyzing the resource utilization, the dependence of different resource units on different types of resources is used as the analysis basis to improve the rationality of resource allocation.

[0063] S23: According to the resource usage ratio of the resource unit to be allocated in each type of resource and the weight corresponding to each type of resource, the average resource utilization rate of the resource unit to be allocated in the allocated node is obtained by weighted average calculation.

[0064] For example, the CPU utilization L (Cpu i ), the memory occupancy rate L(Mem i ), the proportion of disk usage L(Disk i ), for the network bandwidth utilization rate L(Net i); for the usage of these four resources, sort them according to their respective occupancy ratios and assign weights in the ratio of 1:2:3:4; for example, the resource occupancy ratio of resource unit 1 to node 1 is ranked as L(Cpu1)>L(Mem1)>L(Disk1)>L(Net1), then the corresponding weights W(Cpu1), W(Mem1), W(Disk1) and W(Net1) are 4, 3, 2, 1 respectively; the average resource utilization of the node is:

[0065] .

[0066] S24: Determine the average resource utilization rate of the allocation scheme corresponding to the current corrected code string according to the average resource utilization rate of each resource unit to be allocated in the respective allocated node in the current corrected code string.

[0067] According to the average resource utilization of each resource unit on the respective allocated node in the allocation scheme, the overall average resource utilization corresponding to the current corrected code string is determined, that is, the average resource utilization of the cluster after cluster scheduling according to the current corrected code string; the average resource utilization of the allocation scheme is:

[0068] ; Where i represents the resource unit and N represents the node.

[0069] S25: Determine the standard deviation of the current corrected code string according to the average resource utilization corresponding to all generated corrected code strings.

[0070] According to the average resource utilization of different allocation schemes corresponding to different corrected code strings, the standard deviation is used to evaluate each allocation scheme. The average resource utilization of the current allocation scheme L(Clu k ) relative to all allocation options is:

[0071] .

[0072] S26: Taking the standard deviation of the current corrected code string as the individual fitness of the current corrected code string.

[0073] Finally, the calculated standard deviation is used as the individual fitness of the current corrected code string. The genetic algorithm can evaluate the performance of each node in the cluster in solving the optimization problem by calculating the fitness function, ensuring that the algorithm can gradually approach the optimal solution during the iteration process; for example Figure 3As shown in the figure, the genetic algorithm specifically includes selection, crossover and mutation stages. The fitness is used to determine which chromosomes will be used to generate the next generation. The crossover process combines the characteristics of two chromosomes to generate new offspring chromosomes. The mutation operation is used to introduce random changes in the chromosomes to maintain the diversity of the population. Specifically, chromosomes with small variance have a greater probability of being selected. The crossover operation is used to combine the characteristics of two parent chromosomes to generate new offspring chromosomes. The mutation is used to introduce random changes in the chromosomes to maintain the diversity of the population. After the mutation step, a new initial code string is obtained, and a dictionary check is performed again to verify the allocation plan of the resource units.

[0074] Step S14: Filter out the optimal corrected code string according to the individual fitness of all generated corrected code strings, and use the allocation scheme represented by the optimal corrected code string as the cluster resource allocation scheme, so as to perform cluster resource scheduling according to the cluster resource allocation scheme.

[0075] The above-mentioned screening out the optimal corrected code string according to the individual fitness of all the generated corrected code strings may include: sorting the standard deviations corresponding to all the generated corrected code strings; screening out the corrected code string with the smallest standard deviation as the optimal corrected code string. That is, the smaller the standard deviation, the more balanced the load, indicating that the scheduling scheme is better.

[0076] It can be seen that the relationship between the nodes and resource units in this application is represented in a coded form, which can better serve as the input of the improved genetic algorithm; a dictionary check operation is used before the genetic algorithm to check and repair nodes that do not meet the resource requirements of the resource units; with the goal of optimizing cluster resource scheduling, the relationship between the coded nodes and the resource units is used as input, and the optimal resource scheduling plan of the cluster is realized through the improved genetic algorithm, thereby improving resource utilization and achieving load balancing.

[0077] Taking the resource scheduling using Docker containerization technology and Kubernetes container orchestration tools as an example, the resource unit under the Kubernetes cluster is Pod (i.e., the smallest resource scheduling unit); Docker is a popular containerization technology with higher flexibility, fast deployment process, and less system resource occupation; container orchestration tools are an important part of modern cloud computing architecture. Kubernetes can be customized and extended according to different scenarios and requirements, and its scheduling algorithm uses Pod as the smallest resource scheduling unit. In related technologies, all eligible candidate nodes are scored by using the current remaining CPU and memory resources of the node as the scoring criteria, and the algorithm will eventually select the node with the highest score to deploy the corresponding Pod application. However, the current scheduling algorithm only considers the resource utilization of a single node, and does not consider the overall load balancing of the cluster, which will lead to low resource utilization of the cluster; it does not fully consider the diversity of Pod applications, and mainly relies on the two resource indicators of CPU and memory to make scheduling decisions, which may lead to uneven resource allocation; at the same time, the relevant technology uses a unified weight when evaluating all nodes in the cluster, which may cause the resources of some nodes to be over-utilized, while the resources of other nodes are easily neglected; it will affect the overall performance and efficiency of the cluster because resource allocation may not be optimal, especially when facing diverse Pod applications and dynamically changing load demands.

[0078] In this application, as shown in FIG4, by deploying a cluster environment, the allocation scheme code is initialized, a verification dictionary is generated according to the configuration of the cluster nodes and resource units, and the verification is performed using the verification dictionary. Then, the fitness is calculated according to the four indicators of processor utilization, memory occupancy, disk occupancy ratio, and network bandwidth utilization. According to the fitness, a genetic algorithm is used to perform crossover and mutation operations to generate a new initial code until the termination condition of the genetic algorithm is met to obtain the optimal scheduling scheme. It can be seen that the allocation scheme of the genetic algorithm for the cluster is improved by the verification dictionary, which reduces the individuals that do not meet the configuration requirements during the algorithm iteration process, improves the operating efficiency of the scheduling algorithm, and realizes load balancing in the entire cluster; by taking the resource units waiting for the first deployment and the resource units that have been deployed as the resource units to be allocated, multiple resource indicators can be dynamically considered to improve resource utilization and the overall performance of the system.

[0079] As can be seen from the above, in this embodiment, information of multiple resource units to be allocated and nodes in the cluster is obtained, an initial allocation plan for the resource units to be allocated and the nodes is randomly generated, and a first initial code string representing the initial allocation plan is generated; correction and fitness calculation processing is performed on the first initial code string; the correction and fitness calculation processing includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string; according to the first corrected code string, a new initial code string is cyclically generated through a genetic algorithm, and the correction and fitness calculation processing is performed on the new initial code string until the termination condition of the genetic algorithm is met; the optimal corrected code string is screened out according to the individual fitness of all the generated corrected code strings, and the allocation plan represented by the optimal corrected code string is used as the cluster resource allocation plan, so as to perform cluster resource scheduling according to the cluster resource allocation plan. It can be seen that by generating allocation plans for multiple resource units to be allocated and encoding the allocation plans into code strings, the genetic algorithm is used to simulate natural selection and genetic mechanisms to optimize resource allocation, consider the load balancing of the entire cluster, and improve the resource utilization of the entire cluster; and by correcting the initial code string through the verification dictionary, the invalid initial code string is corrected, which improves the speed of screening the optimal resource allocation plan and improves the efficiency of cluster resource scheduling.

[0080] Correspondingly, the embodiment of the present application further discloses a cluster resource scheduling device, as shown in FIG5 , the device includes:

[0081] The first initial code string generating module 11 is used to obtain information of a plurality of resource units to be allocated and nodes in the cluster, randomly generate an initial allocation scheme of the resource units to be allocated and the nodes, and generate a first initial code string representing the initial allocation scheme;

[0082] The correction module 12 is used to perform correction and fitness calculation processing on the first initial code string; the correction and fitness calculation processing includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string;

[0083] A new initial code string generation module 13, used to generate a new initial code string through a genetic algorithm loop according to the first corrected code string, and perform the correction and fitness calculation process on the new initial code string until the termination condition of the genetic algorithm is met;

[0084] The screening module 14 is used to screen out the optimal corrected code string according to the individual fitness of all generated corrected code strings, and use the allocation scheme represented by the optimal corrected code string as the cluster resource allocation scheme to perform cluster resource scheduling according to the cluster resource allocation scheme.

[0085] As can be seen from the above, in this embodiment, information of multiple resource units to be allocated and nodes in the cluster is obtained, an initial allocation plan for the resource units to be allocated and the nodes is randomly generated, and a first initial code string representing the initial allocation plan is generated; correction and fitness calculation processing is performed on the first initial code string; the correction and fitness calculation processing includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string; according to the first corrected code string, a new initial code string is cyclically generated through a genetic algorithm, and the correction and fitness calculation processing is performed on the new initial code string until the termination condition of the genetic algorithm is met; the optimal corrected code string is screened out according to the individual fitness of all the generated corrected code strings, and the allocation plan represented by the optimal corrected code string is used as the cluster resource allocation plan, so as to perform cluster resource scheduling according to the cluster resource allocation plan. It can be seen that by generating allocation plans for multiple resource units to be allocated and encoding the allocation plans into code strings, the genetic algorithm is used to simulate natural selection and genetic mechanisms to optimize resource allocation, consider the load balancing of the entire cluster, and improve the resource utilization of the entire cluster; and by correcting the initial code string through the verification dictionary, the invalid initial code string is corrected, which improves the speed of screening the optimal resource allocation plan and improves the efficiency of cluster resource scheduling.

[0086] In some specific embodiments, the verification dictionary can be specifically generated according to the deployment type of each node relative to each resource unit to be allocated; the deployment type includes allowed deployment and prohibited deployment; the deployment type is determined based on the resource requirements of each resource unit to be allocated and the free resources of each node in the cluster.

[0087] In some specific embodiments, the correction module 12 may specifically include:

[0088] A verification unit, configured to verify the allocation of the resource units to be allocated in the first initial code on the nodes using the verification dictionary;

[0089] A screening unit is used to screen out target resource units to be allocated that do not conform to the verification dictionary, reallocate nodes to the target resource units to be allocated according to the verification dictionary, and regenerate a code string to obtain the first corrected code string.

[0090] In some specific embodiments, the new initial code string generation module 13 may specifically include:

[0091] A second initial code string generating unit, configured to generate a second initial code string using a genetic algorithm based on the first corrected code string and the fitness of each chromosome in the first corrected code string; the chromosome in the first corrected code string represents the allocation relationship between each resource unit to be allocated and each corresponding node;

[0092] The correction and fitness calculation unit is used to perform the correction and fitness calculation processing on the second initial code string.

[0093] In some specific embodiments, the cluster resource scheduling device may specifically include:

[0094] A resource usage ratio determination unit, used to determine the resource usage ratio of different types of resources of the resource unit to be allocated on the allocated node in the current corrected code string;

[0095] A weight configuration unit, used to sort the usage proportions of each type of resource corresponding to the resource unit to be allocated according to the numerical value, and configure a weight for each type of resource; wherein, the higher the numerical value of the resource usage proportion, the higher the weight corresponding to the type of resource, and the lower the numerical value of the resource usage proportion, the lower the weight corresponding to the type of resource;

[0096] A node resource average utilization rate determination unit, configured to obtain the resource average utilization rate of the resource unit to be allocated in the allocated node by calculating the weighted average value according to the resource usage ratio of the resource unit to be allocated in each type of resource and the weight corresponding to each type of resource;

[0097] A scheme resource average utilization unit, used to determine the resource average utilization of the allocation scheme corresponding to the current corrected code string according to the resource average utilization of each of the to-be-allocated resource units at the respective allocated nodes in the current corrected code string;

[0098] A standard deviation determining unit, configured to determine a standard deviation of the current corrected code string according to an average resource utilization rate corresponding to all generated corrected code strings;

[0099] The individual fitness determination unit is used to use the standard deviation of the current corrected code string as the individual fitness of the current corrected code string.

[0100] In some specific embodiments, the resource usage ratio may specifically include processor utilization, memory occupancy, disk occupancy ratio, and network bandwidth utilization.

[0101] In some specific embodiments, the screening module 14 may specifically include:

[0102] A sorting unit, used for sorting the standard deviations corresponding to all the generated corrected code strings;

[0103] The optimal corrected code string determining unit is used to select the corrected code string with the smallest standard deviation as the optimal corrected code string.

[0104] Furthermore, an embodiment of the present application also discloses an electronic device, as shown in FIG6 . The content in the figure cannot be regarded as any limitation on the scope of use of the present application.

[0105] FIG6 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the cluster resource scheduling method disclosed in any of the aforementioned embodiments.

[0106] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0107] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including individual fitness, etc. The storage method can be temporary storage or permanent storage.

[0108] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the cluster resource scheduling method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0109] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the cluster resource scheduling method steps disclosed in any of the aforementioned embodiments are implemented.

[0110] Furthermore, an embodiment of the present application also discloses a computer program product, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the cluster resource scheduling method disclosed in any of the aforementioned embodiments are implemented.

[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0112] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0113] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0114] The above is a detailed introduction to a cluster resource scheduling method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A cluster resource scheduling method, characterized in that: include: Acquire information of a plurality of resource units to be allocated and nodes in a cluster, randomly generate an initial allocation scheme of the resource units to be allocated and the nodes, and generate a first initial code string representing the initial allocation scheme; Performing correction and fitness calculation processing on the first initial code string; The correction and fitness calculation process includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string; Generate a new initial code string through a genetic algorithm loop according to the first corrected code string, and perform the correction and fitness calculation process on the new initial code string until the termination condition of the genetic algorithm is met; The optimal corrected code string is screened out according to the individual fitness of all generated corrected code strings, and the allocation scheme represented by the optimal corrected code string is used as the cluster resource allocation scheme, so as to perform cluster resource scheduling according to the cluster resource allocation scheme.

2. The cluster resource scheduling method according to claim 1, characterized in that: The verification dictionary is generated according to the deployment type of each node relative to each resource unit to be allocated; the deployment type includes allowed deployment and prohibited deployment; The deployment type is determined according to the resource requirements of each of the resource units to be allocated and the idle resources of each node in the cluster.

3. The cluster resource scheduling method according to claim 2, characterized in that: The method of correcting the first initial code string by using a pre-built verification dictionary to obtain a first corrected code string includes: Using the verification dictionary to verify the allocation of the resource units to be allocated in the first initial code on the nodes; Target resource units to be allocated that do not conform to the verification dictionary are screened out, nodes are reallocated for the target resource units to be allocated according to the verification dictionary, and a code string is regenerated to obtain the first corrected code string.

4. The cluster resource scheduling method according to claim 1, characterized in that: The step of cyclically generating a new initial code string through a genetic algorithm according to the first corrected code string comprises: Based on the first corrected code string and the fitness of each chromosome in the first corrected code string, a second initial code string is generated by using a genetic algorithm; the chromosome in the first corrected code string represents the allocation relationship between each resource unit to be allocated and each corresponding node; The correction and fitness calculation process is performed on the second initial code string.

5. The cluster resource scheduling method according to any one of claims 1 to 4, characterized in that: The process of determining the individual fitness includes: Determine the resource usage ratio of different types of resources of the resource unit to be allocated on the allocated node in the current corrected code string; Sort the usage proportions of each resource corresponding to the resource unit to be allocated according to the numerical values, and configure weights for each type of resource; wherein, the higher the numerical value of the resource usage proportion, the higher the weight corresponding to the type of resource, and the lower the numerical value of the resource usage proportion, the lower the weight corresponding to the type of resource; According to the resource usage ratio of the resource unit to be allocated in each type of resource and the weight corresponding to each type of resource, the average resource utilization rate of the resource unit to be allocated in the allocated node is obtained by weighted average calculation; According to the average resource utilization rate of each resource unit to be allocated in the current corrected code string at the respective allocated node, determining the average resource utilization rate of the allocation scheme corresponding to the current corrected code string; Determine the standard deviation of the current corrected code string according to the average resource utilization rate corresponding to all generated corrected code strings; The standard deviation of the current corrected code string is used as the individual fitness of the current corrected code string.

6. The cluster resource scheduling method according to claim 5, characterized in that: The resource usage ratio includes processor utilization, memory occupancy, disk occupancy ratio, and network bandwidth utilization.

7. The cluster resource scheduling method according to claim 5, characterized in that: The step of selecting the optimal corrected coding string according to the individual fitness of all generated corrected coding strings comprises: Sort the standard deviations corresponding to all the corrected code strings generated; The corrected code string with the smallest standard deviation is selected as the optimal corrected code string.

8. A cluster resource scheduling device, characterized in that: include: A first initial code string generation module, used to obtain information of a plurality of resource units to be allocated and nodes in a cluster, randomly generate an initial allocation scheme of the resource units to be allocated and the nodes, and generate a first initial code string representing the initial allocation scheme; A correction module, used for performing correction and fitness calculation processing on the first initial code string; The correction and fitness calculation process includes: using a pre-constructed verification dictionary to correct the first initial code string to obtain a first corrected code string, and determining the individual fitness of the first corrected code string; A new initial code string generation module, used to generate a new initial code string through a genetic algorithm loop according to the first corrected code string, and perform the correction and fitness calculation process on the new initial code string until the termination condition of the genetic algorithm is met; The screening module is used to screen out the optimal corrected code string according to the individual fitness of all generated corrected code strings, and use the allocation scheme represented by the optimal corrected code string as the cluster resource allocation scheme to perform cluster resource scheduling according to the cluster resource allocation scheme.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the cluster resource scheduling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the cluster resource scheduling method as described in any one of claims 1 to 7 is implemented.

Images