A method and device for allocating computing resources based on deep reinforcement learning

Abstract

This invention proposes a computing resource allocation method and apparatus based on deep reinforcement learning. The computing resource allocation method based on deep reinforcement learning can extract feature information such as physical machines and virtual machines in the system, perform feature extraction by using an encoder, and make decisions in combination with a deep reinforcement learning model. Under complex and diverse constraints, it can meet users' online requests in real time, effectively utilize CPU and memory resources in the resource pool, and further optimize the model effect through action masking to improve the system resource utilization rate.

Description

Technical Field

[0001] This invention relates to the field of resource management technology in cloud computing, and particularly to a computing resource allocation method, apparatus, electronic device, computer-readable storage medium, and computer program product based on deep reinforcement learning. Background Technology

[0002] Cloud computing, as one of the core technologies in today's information technology field, has formed a vast and complex ecosystem of technologies, products, and services. It provides flexible, on-demand computing resources, enabling users to acquire powerful computing capabilities at a lower cost, thus driving enterprise digital transformation and innovation. However, with the continuous expansion of cloud computing services, how to efficiently allocate and manage these resources has become a crucial issue, especially against the backdrop of the rapid expansion and increasingly fierce competition in the global Infrastructure as a Service (IaaS) market. By optimizing resource utilization, cloud service providers can significantly reduce operating costs, and the placement of virtual machines is one of the core aspects of this process.

[0003] In cloud computing environments, IaaS providers offer computing services to users through resources such as virtual machines, storage, and networks. A key feature of this model is that users can dynamically adjust resource usage based on actual needs and only pay for the resources actually consumed. This flexibility makes cloud computing the preferred solution for enterprises to optimize IT costs and accelerate business innovation. However, with the widespread adoption of cloud computing, cloud service providers face increasing pressure in resource management, especially in resource allocation and scheduling. Effective resource allocation not only affects the performance of cloud services but also directly impacts the cost structure of cloud service providers. The placement of each virtual machine (VM) involves the rational use of computing, storage, and network resources; slight missteps can lead to resource waste and even affect the user's service experience. More importantly, with the diversification of cloud computing resource demands, different applications and users have varying resource requirements. How to maximize resource utilization while ensuring service quality has become a core challenge in cloud computing resource management. The VM Placement Problem (VMP), as an important component of resource allocation, determines which physical machine (PM) each VM should be assigned to, thereby maximizing resource utilization, reducing energy consumption, and improving service quality. In practical applications, the placement of virtual machines requires consideration of various complex factors, such as computing resource consumption, network latency, communication bandwidth, and fault domain isolation. Therefore, the virtual machine placement problem plays a crucial role in cloud computing resource management, and its complexity leads it to be classified as an NP-hard problem.

[0004] The virtual machine placement problem, a typical NP-hard problem, faces its core challenge in multi-objective optimization. First, resources in a cloud computing environment are finite and heterogeneous, with different physical servers possessing varying computing, storage, and network resources. Therefore, rationally matching the needs of virtual machines with the capabilities of physical servers is a complex decision-making problem. Second, virtual machine placement must satisfy a series of constraints. For example, some applications are highly latency-sensitive, and communication between virtual machines must remain within a low latency range; thus, virtual machine placement needs to consider the network topology between physical servers. Furthermore, to improve system fault tolerance, virtual machines need to be distributed across different fault domains to prevent single points of failure from causing application unavailability. Moreover, the virtual machine placement problem involves dynamism. The cloud computing environment is a highly dynamic system; user needs are constantly changing, and virtual machines can be frequently created, destroyed, or migrated. This means that virtual machine placement is not only a static problem but also requires consideration of runtime adjustments and optimizations. For example, when the resource load on some physical servers is too high, it may be necessary to migrate some virtual machines to other servers to avoid performance bottlenecks or resource overload. It is precisely because of these complex constraints and the demands of a dynamic environment that the virtual machine placement problem is classified as an NP-hard problem. Against this backdrop, how to develop effective algorithms to solve the virtual machine placement problem has become a hot topic in cloud computing research.

[0005] Currently, significant progress has been made in the research of virtual machine placement problems. Traditional methods can be broadly categorized into heuristic algorithms and linear programming-based optimization algorithms. However, these methods often have limitations when dealing with large-scale, complex cloud computing environments. Heuristic algorithms, such as greedy algorithms, simulated annealing, and genetic algorithms, can provide relatively effective approximate solutions to the virtual machine placement problem, but these algorithms typically rely on specific initial conditions or preset rules, making them prone to getting trapped in local optima. Especially in large-scale cloud computing environments, heuristic algorithms have slow convergence speeds, and the quality of solutions is difficult to guarantee. Furthermore, heuristic algorithms have poor adaptability to dynamic environments; once cloud resources change, the entire placement scheme often needs to be recalculated, leading to reduced resource utilization. Linear programming-based optimization methods can theoretically provide optimal solutions, but their computational complexity increases rapidly with the problem size. In large-scale cloud computing environments, the number of virtual machines and constraints are very complex, and the solution process may consume a large amount of computational resources and time, making it difficult to meet the real-time requirements of practical applications. In addition, linear programming algorithms typically assume that the problem is static, making it difficult to adapt to the dynamic migration and real-time adjustment needs of virtual machines. With the diversification of cloud computing applications, traditional virtual machine placement algorithms often fail to meet the needs of different users and applications simultaneously. For example, some applications have high demands for computing resources, while others are more concerned about network latency or communication bandwidth. Traditional algorithms struggle to accommodate these diverse needs, resulting in unsatisfactory resource allocation.

[0006] To address the limitations of traditional methods, Deep Reinforcement Learning (DRL) has gradually emerged as a new approach for solving the virtual machine placement problem in recent years, thanks to the rapid development of artificial intelligence technology. Compared with traditional methods, DRL boasts strong adaptability, a powerful ability to handle complex constraints, and high scalability. In the virtual machine placement problem, DRL algorithms can adjust the placement scheme in real time according to the actual situation of the cloud computing environment, without the need for pre-defined rules. This allows DRL to exhibit extremely strong adaptability in dynamic environments; whether it's the creation, destruction, or migration of virtual machines, DRL can respond quickly and make optimal decisions. However, most current reinforcement learning algorithms suffer from poor training results and unsatisfactory model performance due to issues such as inefficient information acquisition and ineffective output actions.

[0007] In summary, existing computing resource allocation techniques are insufficient to effectively address the virtual machine placement problem in this scenario, and existing deep reinforcement learning techniques have limited research on this type of issue, failing to fully utilize system resources and improve system performance. Therefore, it is necessary to improve existing technologies, further optimize model performance, and enhance system resource utilization. Summary of the Invention

[0008] To address the shortcomings of existing technologies, such as Figure 4 As shown, this invention proposes a computational resource allocation method based on deep reinforcement learning, which includes:

[0009] The initial step is to obtain a cloud computing resource pool that includes multiple servers, and divide the cloud computing resource pool into: network domain, deployment unit, rack and server. The network domain contains multiple deployment units, and the deployment unit is an independent unit in the network domain. The deployment unit contains multiple racks, and each rack is connected to the core network of the deployment unit by one or more network switches, and the rack contains multiple servers.

[0010] The data collection steps involve collecting the current status information of the cloud computing resource pool, including: hardware performance indicators of each server, status information of virtual machines running on the servers, constraint information of the cloud computing resource pool in operation, and request information of computing resources required by each virtual machine.

[0011] The analysis steps involve inputting the feature representation of the state information into the reinforcement learning agent, and analyzing the state of each server by loading a pre-trained deep neural network to obtain the probability distribution matrix of each server being selected by the virtual machine in the request information.

[0012] The allocation step involves determining the compliance of each action corresponding to an element in the probability distribution matrix based on the hardware performance metric information and the constraint information. Compliant actions are set to a positive number, while non-compliant actions are set to 0, resulting in an action matrix with the same dimension as the probability distribution matrix. This action matrix is ​​then multiplied by the probability distribution matrix to obtain the final action probability matrix. Based on this final action probability matrix, the action corresponding to the largest element in the final action probability matrix is ​​selected for each virtual machine as the current computing resource allocation strategy. A server is then allocated to the virtual machine according to this computing resource allocation strategy.

[0013] The aforementioned computational resource allocation method based on deep reinforcement learning includes the following acquisition step:

[0014] Collect actual server operating information, including the server's remaining computing resources; obtain virtual machine status information, including: virtual machine identifier, virtual machine associated constraint identifier, and virtual machine placement location; obtain pre-set constraint information and request information, including affinity type and non-affinity type of constraint information to ensure that the relationship between virtual machines is considered during resource allocation; request information includes the number of requested virtual machines, the amount of computing resources required by each virtual machine, and the constraint identifiers to be followed.

[0015] The analytical steps include:

[0016] The reinforcement learning agent uses a fully connected network structure. The virtual machine state information and the constraint information are represented by fixed-length arrays. Uncreated virtual machines and constraints in the arrays are filled with specified characters. All state information is merged and concatenated. The concatenated state information is input into an encoder composed of multi-layer neural networks. The encoder performs dimensionality reduction, redundant information removal and feature extraction on the input state information through nonlinear transformation to generate a feature representation of the state information.

[0017] Based on this feature representation, the reinforcement learning agent assigns a probability value to each possible operation, representing its tendency to perform that operation, until all virtual machines currently requesting the operation have been assigned probability values, thus obtaining the probability distribution matrix.

[0018] The allocation step includes: judging the action corresponding to each element in the probability distribution matrix, judging the actions of servers with remaining computing resources less than the computing resources required by the virtual machine as non-compliant, and judging the actions of servers that do not meet the virtual machine constraints as non-compliant.

[0019] The computational resource allocation method based on deep reinforcement learning includes the following steps: creating a new initial matrix with the same dimension as the probability distribution matrix; performing compliance judgment on the action corresponding to each element in the probability distribution matrix; setting the value in the initial matrix corresponding to the action of the server with insufficient CPU or memory resources to 0; setting the value in the initial matrix corresponding to the action of the server that does not meet the virtual machine constraints to 0; setting the values ​​in the other initial matrices to 1; and saving the current initial matrix as the action matrix after all elements in the initial matrix have been set.

[0020] The computational resource allocation method based on deep reinforcement learning, wherein the request information includes: creating constraints, and / or batch creating virtual machines, and / or deleting virtual machines;

[0021] The creation constraint includes the identification number and category of the newly created constraint;

[0022] This batch creation of virtual machines includes the type and quantity of virtual machines to be created, as well as the constraints that must be met;

[0023] This option deletes a virtual machine, including the number of the specified virtual machine to be deleted.

[0024] like Figure 5 As shown, the present invention also proposes a computing resource allocation device based on deep reinforcement learning, comprising:

[0025] The initial module acquires a cloud computing resource pool including multiple servers and divides the cloud computing resource pool into: network domain, deployment unit, rack and server. The network domain contains multiple deployment units, which are independent units in the network domain. Each deployment unit contains multiple racks, and each rack is connected to the core network of the deployment unit by one or more network switches. Each rack contains multiple servers.

[0026] The data acquisition module collects the current status information of the cloud computing resource pool, including: hardware performance indicators of each server, status information of virtual machines running on the server, constraint information of the cloud computing resource pool, and request information of computing resources required by each virtual machine.

[0027] The analysis module inputs the feature representation of the state information into the reinforcement learning agent, and analyzes the state of each server by loading a pre-trained deep neural network to obtain the probability distribution matrix of each server being selected by the virtual machine in the request information.

[0028] The allocation module, based on the hardware performance information and the constraint information, determines the compliance of the action corresponding to each element in the probability distribution matrix. Compliant actions are set to a positive number, and non-compliant actions are set to 0, resulting in an action matrix with the same dimension as the probability distribution matrix. The action matrix is ​​multiplied by the probability distribution matrix to obtain the final action probability matrix. Based on the final action probability matrix, the module selects the action corresponding to the largest element in the final action probability matrix for each virtual machine as the current computing resource allocation strategy, and allocates servers to the virtual machines according to the computing resource allocation strategy.

[0029] The aforementioned deep reinforcement learning-based computing resource allocation device, wherein the acquisition module includes:

[0030] Collect actual server operating information, including the server's remaining computing resources; obtain virtual machine status information, including: virtual machine identifier, virtual machine associated constraint identifier, and virtual machine placement location; obtain pre-set constraint information and request information, including affinity type and non-affinity type of constraint information to ensure that the relationship between virtual machines is considered during resource allocation; request information includes the number of requested virtual machines, the amount of computing resources required by each virtual machine, and the constraint identifiers to be followed.

[0031] This analysis module includes:

[0032] The reinforcement learning agent uses a fully connected network structure. The virtual machine state information and the constraint information are represented by fixed-length arrays. Uncreated virtual machines and constraints in the arrays are filled with specified characters. All state information is merged and concatenated. The concatenated state information is input into an encoder composed of multi-layer neural networks. The encoder performs dimensionality reduction, redundant information removal and feature extraction on the input state information through nonlinear transformation to generate a feature representation of the state information.

[0033] Based on this feature representation, the reinforcement learning agent assigns a probability value to each possible operation, representing its tendency to perform that operation, until all virtual machines currently requesting the operation have been assigned probability values, thus obtaining the probability distribution matrix.

[0034] The allocation module includes: judging the action corresponding to each element in the probability distribution matrix, judging the action corresponding to the server whose remaining computing resources are less than the computing resources required by the virtual machine as non-compliant, and judging the action corresponding to the server that does not meet the virtual machine constraints as non-compliant.

[0035] The computing resource allocation device based on deep reinforcement learning includes the following: creating a new initial matrix with the same dimension as the probability distribution matrix; performing compliance judgment on the action corresponding to each element in the probability distribution matrix; setting the value in the initial matrix corresponding to the action of the server with insufficient CPU or memory resources to 0; setting the value in the initial matrix corresponding to the action of the server that does not meet the virtual machine constraints to 0; setting the value in the other initial matrices to 1; and saving the current initial matrix as the action matrix after all elements in the initial matrix have been set.

[0036] The request information includes: creating constraints, and / or creating virtual machines in bulk, and / or deleting virtual machines;

[0037] The creation constraint includes the identification number and category of the newly created constraint;

[0038] This batch creation of virtual machines includes the type and quantity of virtual machines to be created, as well as the constraints that must be met;

[0039] This option deletes a virtual machine, including the number of the specified virtual machine to be deleted.

[0040] The present invention also proposes an electronic device, including the aforementioned deep reinforcement learning-based computing resource allocation device, which may be connected to an information display device, the information display device being used to display the computing resource allocation strategy using user-set display parameters, attributes, or through an artificial intelligence model.

[0041] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep reinforcement learning-based computational resource allocation method.

[0042] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the deep reinforcement learning-based computational resource allocation method.

[0043] As can be seen from the above solutions, the advantages of the present invention are:

[0044] In summary, the computational resource allocation method based on deep reinforcement learning in this invention can extract feature information such as physical machines and virtual machines in the system. It uses an encoder for feature extraction and combines it with a deep reinforcement learning model for decision-making. Under complex and diverse constraints, it can meet users' online requests in real time, effectively utilize CPU and memory resources in the resource pool, and further optimize model performance through action masks, thereby improving system resource utilization. Compared with traditional heuristic algorithms, the placement success rate is improved by 85.3%, resource utilization by 35.4%, and placement response time by 58.4%. Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating an application scenario of a computational resource allocation method for deep reinforcement learning according to an example of the present invention.

[0046] Figure 2 A flowchart illustrating a computational resource allocation method for deep reinforcement learning according to an example of the present invention;

[0047] Figure 3 A flowchart of action mask calculation in a computational resource allocation method for deep reinforcement learning according to an example of the present invention;

[0048] Figure 4 This is a flowchart of the method of the present invention;

[0049] Figure 5 This is a block diagram of the device of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0051] Figure 7 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0052] Figure 8 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0053] Figure label:

[0054] A - First electronic device;

[0055] B- A computational resource allocation device based on deep reinforcement learning;

[0056] C-Data acquisition equipment;

[0057] D-Information display device;

[0058] 1000 - Second electronic device;

[0059] Ⅰ-Computational Unit;

[0060] II-ROM;

[0061] III-RAM;

[0062] N-bus;

[0063] V-Interface;

[0064] VI - Input Unit;

[0065] VII - Output Unit;

[0066] VIII - Storage medium;

[0067] IX - Communication Unit. Detailed Implementation

[0068] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0069] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0070] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0071] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0072] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0073] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0074] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0075] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0076] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0077] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0078] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0082] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The purpose of this invention is to overcome the above-mentioned technical defects and provide a method for allocating computing resources based on deep reinforcement learning.

[0084] The method includes the following steps:

[0085] Step 1: Considering the strong correlation and high dimensionality of data information in the system, at each unit of time, by calling the system interface, at least four aspects of status information at the current time are obtained from various complex information in the system: physical machine status information, covering the operating status and performance indicators of hardware resources; virtual machine status information, describing the resource allocation and load in the virtualization environment; constraint information, referring to various restrictions and rules in system operation; and request information, reflecting the specific needs of users or tasks for system resources.

[0086] Step 2: After processing and merging the extracted state information, ensure the consistency and readability of various state data. Encode the merged data through an encoder to extract its deep feature representation, which is then used as the input to the reinforcement learning agent. The agent, as part of the decoder, loads a pre-trained deep neural network (e.g., a multilayer perceptron) to analyze the state information (hardware information and virtual machine information) of each server, thereby obtaining the probability distribution of each server being selected.

[0087] The reinforcement learning agent refers to the decision-making entity, which learns the most strategic approach by perceiving the state of the environment, taking actions, and receiving rewards during the interaction with the environment. It can be determined according to the specific task requirements and the actual situation, such as the actor-critic model.

[0088] Step 3: Within the current unit of time, calculate the mask for each action. First, initialize a matrix of the same size as the dimension of the action space. Then, judge each action. Set a positive number for actions that meet the current environmental conditions and set 0 for those that do not meet the conditions. This gives the action mask matrix. Then, multiply the mask by the probability distribution value to get the final action probability distribution. This achieves the effect that actions that cannot be selected will definitely not be selected. Finally, select the actions according to the probability distribution, create the corresponding physical machine, and provide feedback information.

[0089] In one instance, step 1 includes:

[0090] Step 11: Collect the actual operating information of the physical machine, including all data related to Non-Uniform Memory Access (NUMA), namely CPU usage and memory usage, as well as detailed hardware information of the physical machine, including server, rack, deployment unit (Pod) and network domain. It is also necessary to obtain the remaining computing resources of the physical machine, especially the number of currently available CPU cores and memory capacity, in order to provide accurate basic data for subsequent resource scheduling.

[0091] Step 12: Obtain virtual machine status information, including the unique identifier of the virtual machine itself, the unique identifier of its associated constraints, and the specific location information, such as the server, rack, pod, and network to which it belongs.

[0092] Step 13: Obtain the set constraint information and request information. The constraint information includes affinity type and non-affinity type to ensure that the relationship between virtual machines is considered during resource allocation. The request information includes the number of virtual machines requested, the amount of CPU and memory resources required by each virtual machine, and the constraint ID to be followed.

[0093] In one instance, step 2 includes:

[0094] Step 21: Further process the state information obtained in Step 1. Since the reinforcement learning model of this method uses a fully connected network structure, the input dimension must be a fixed value. Therefore, the virtual machine information and constraint information in Step 12 are represented by fixed-length arrays. For virtual machines and constraints that have not yet been created, they will be filled with -1. Thus, the length of the input can be fixed. Then, all the state information in Step 1 is merged and concatenated to obtain the state information of the input agent.

[0095] Step 22: After completing the splicing of state information, the state information obtained in step 21 is input into the encoder composed of a multi-layer neural network. The encoder performs dimensionality reduction, redundant information removal and feature extraction on the input high-dimensional data through nonlinear transformation to generate a compact feature vector representation.

[0096] Step 23: The feature vector obtained in Step 22 is input into the agent. The agent, based on its trained neural network model, calculates the corresponding action probability distribution through inference. This process, based on the features of the input state, assigns a probability value to each possible operation, representing its tendency to perform that operation, until all currently requested virtual machines have been placed.

[0097] In one instance, step 3 includes:

[0098] Step 31: Calculate the action mask. First, initialize a matrix of the same size as the action space dimension. Then, judge each action and set the value in the matrix corresponding to the action of the server with insufficient CPU or memory resources to 0. Set the value in the matrix corresponding to the action of the server that does not meet the virtual machine constraints to 0. Set the value in the matrix of the other actions to 1. This generates the action mask.

[0099] Step 32: Multiply the generated action mask by the action probability distribution generated in step 22 to obtain the probability distribution of all actions that meet the conditions. Then, use the actions according to the probability distribution as the final output, feed the output actions back into the system to realize the creation of the virtual machine, and then obtain the latest system status information.

[0100] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0101] Figure 1 This paper illustrates an application scenario of the present invention, considering placing virtual machines in a resource pool composed of servers from one or more data centers. The size of the resource pool is fixed, and no additional resources are added or removed during operation. The resources of each server are represented by CPU and memory capacity. Since all mainstream servers currently use a Non-Uniform Memory Access (NUMA) architecture, each server is divided into multiple NUMA nodes, and each node holds a portion of the server's CPU and memory resources. Therefore, the resources of each node are not necessarily the same. The resource pool consists of four layers: network domain, Pod, rack, and server. A network domain is the highest level of the entire resource pool. It can cover multiple data centers or large-scale computing clusters, providing the infrastructure for network interconnection and resource scheduling. A network domain contains multiple Pods. A Pod typically represents a data center in a physical location or a partition within it. It is an independent unit in a network domain, possessing relatively independent network and computing resources. It can be viewed as a modular unit in a data center. A Pod contains multiple racks. A rack is a physical structure within a Pod, typically used to house multiple servers. Each rack is connected to the Pod's core network by one or more network switches to provide computing, storage, and communication capabilities. A rack contains multiple servers. A server is the basic computing unit in a data center, providing specific computing and storage resources.

[0102] Users can create virtual machines according to the specified virtual machine type. The characteristics of each virtual machine type are the number of NUMA nodes and the resource (CPU and memory) usage of each NUMA node. Users can apply to create multiple virtual machines of the same type at the same time, i.e., batch creation. Users can request to delete any previously created virtual machine at any time. The time of deletion of a virtual machine is unknown and may be initiated at any time.

[0103] Each virtual machine is created within constraints, which are mainly divided into the following types:

[0104] 1. No restrictions: It can be placed as long as the server has sufficient CPU and memory resources;

[0105] 2. Rack affinity: All virtual machines belonging to the same constraint must be placed in the same rack;

[0106] 3. Pod affinity: All virtual machines belonging to the same constraint must be placed in the same Pod;

[0107] 4. Network domain affinity: All virtual machines belonging to the same constraint must be placed in the same network domain;

[0108] 5. Server incompatibility: Virtual machines belonging to the same constraint cannot be placed on the same server;

[0109] 6. Rack affinity: Virtual machines belonging to the same constraint cannot be placed in the same rack;

[0110] 7. Rack affinity + Server in affinity: All virtual machines belonging to the same constraint must be placed in the same rack and cannot be placed in the same server;

[0111] 8. Network domain affinity + rack inaffinity: All virtual machines belonging to the same constraint must be placed in the same network domain and cannot be placed in the same rack.

[0112] Therefore, users can make three types of requests:

[0113] 1. Create constraints: Define a unique identifier and category for the constraint. This operation is performed directly within the system.

[0114] 2. Batch creation of virtual machines: Specify the type, quantity, and constraints of the virtual machines. The system needs to derive a specific placement strategy for the virtual machines based on this method and then create the virtual machines.

[0115] 3. Delete virtual machine: You can delete the virtual machine immediately by specifying its unique identifier. This operation is performed directly in the system.

[0116] Figure 2 The specific steps for placing virtual machines online according to the present invention are shown below:

[0117] Step S310: Obtain system information and perform partial preprocessing on the information.

[0118] In this step, the actual operating information of the physical machine is first collected, then the status information of the virtual machine is obtained, and finally the constraint information and user request information are obtained. The specific steps are as follows:

[0119] S311: Collects actual operating information of the physical machine. For the i-th NUMA node, this includes information such as CPU usage (ucpu). i Memory usage (umem) i Detailed hardware information, including the server bserver. i rack i The Pod it belongs to (bpod_i) and the network domain it belongs to (bdomain) i The remaining computing resources of the physical machine, including the currently available CPU cores (rcpu). i With current available memory rmem i Therefore, the operating status information of the physical machine is represented as follows:

[0120] snuma i =[ucpu i ,umem i ,bserver i ,brack i ,bpod i ,bdomain i ,rcpu i ,rmem i ]

[0121] S312: Obtain virtual machine status information. For the j-th virtual machine, this includes: unique identifier vid. j The unique identifier of the constraint, vmpg j The physical information it belongs to, and the server it resides on, pserver. j The rack in which it is located j Podppod j and the network domain pdomain j Therefore, the virtual machine status information is represented as follows:

[0122] svm j =pvid j pgid j ,pserver j ,prack j ppod j ,pdomain j ,deleted j ]

[0123] S313: Obtain constraint information and request information. For the k-th constraint, the information includes: the constraint's unique identifier pgid. k Type pgtype k ∈[1,8]. Therefore, the constraint information is represented as follows:

[0124] spg k =[pgid k ,pgtype k ]

[0125] Since user requests for virtual machine deletion and constraint creation directly invoke system interfaces and are not involved in this method, we only consider the information from the user's virtual machine creation request. The t-th creation request includes the following information: the number of virtual machines requested in the batch (anum). t CPU required per virtual machine (ACPU) t The number of NUMA nodes required per virtual machine (anuma) t The amount of memory required for each virtual machine (amem) t and the unique identifier apg of the constraints to be followed. t Therefore, the application information is represented as follows:

[0126] sapp t =[anum t ,anuma t ,acpu t ,amem t ,apg t ]

[0127] Step S320: Process and merge the acquired state information, input it into the encoder, and then input the feature information extracted from the encoder into the agent. The agent outputs the preliminary action distribution based on the trained neural network.

[0128] In this step, the virtual machine state information and constraint information first need to be further processed. Due to limited system resources, there is a certain value, which is the maximum number of virtual machines that can be placed. This value is set as VK. Similarly, since at least one virtual machine is bound in a constraint, there is also a maximum value, which is set as PK. The existing state information and constraint information are then padded to make its length VK and PK. Then, the physical machine information, virtual machine state information, constraint information, and request information are merged. The specific merging process is described in S322 to obtain the final state information. The state information is then input into the pre-trained agent. The agent infers based on the neural network model and obtains the corresponding action distribution as the output. The steps for training the neural network are as follows:

[0129] S321: Further process the virtual machine state information and constraint information. Assuming there are J virtual machines that have not been deleted and K constraints, the virtual machine state information is defined as follows:

[0130] SVM = [svmj ,-1,-1,…,-1],j∈{1,2,…,J}

[0131] The length of the -1 padded at the end is VK-J, and the state information of the constraint condition is defined as follows:

[0132] SPG = [spg] k ,-1,-1,…,-1],k∈{1,2,…,K}

[0133] The length of the -1 padding is PK-K.

[0134] S322: Merge various types of state information. Assuming there are I NUMA nodes, the state information merging calculation method is as follows for the T-th request:

[0135] STATE T =[SNUMA T SVM T SPG T SAPP T ]

[0136] The calculation methods for SVM and SPG are described in step S321, and the calculation methods for SNUMA and SAPP are as follows:

[0137] SNUMA = [snuma] i ], i∈{1,2,…I}

[0138] SAPP = [sapp] t ],t∈{1,2,…T}

[0139] S323: STATE T The data is input into the encoder, which consists of a multilayer perceptron used to perform nonlinear transformations, reduce the dimensionality of high-dimensional data, and extract features to obtain feature information.

[0140] S324: Set the agent's actions and reward values, where the action value ranges from [0, S-1], and S represents the number of servers, corresponding to the unique identifier of each selectable server. Since the goal of this method is to utilize resources as much as possible and satisfy as many virtual machine creation requests as possible, the reward value is designed as follows: For each successful creation of a virtual machine, return a reward value of 1, and repeat steps S321, S322, and S323 until all requirements are met; otherwise, return a reward value of 0 and end the current training round.

[0141] S325: The agent is designed with three fully connected layers. The agent model is trained according to the PPO algorithm. The PPO algorithm is a reinforcement learning algorithm proposed by OpenAI. It is a relatively mature model training algorithm, so this method will not be described in detail.

[0142] Step S330: Calculate the action mask based on the current system information, further obtain the action probability distribution that conforms to the rules, and create a virtual machine based on the distribution to update the system information.

[0143] In this step, to further improve the agent's performance and increase training speed, an action mask needs to be calculated using system information to remove unsuitable server selections. Then, the final placement action is calculated based on the action mask, a virtual machine is created in the system, system information is updated, and the latest placement decision is calculated according to steps S310 and S320. The steps for calculating the action mask and obtaining the final action probability are as follows:

[0144] S331: Initialize a matrix with the same size as the action space, i.e., a sequence of length S after flattening, as the initial case for the action mask, and initialize its value to 1.

[0145] S332: Calculate servers that do not meet resource requirements, initially update the action mask, and filter out servers with insufficient remaining resources based on the CPU and memory requirements of the requested virtual machines. Set the action mask matrix of their corresponding servers to 0. The specific calculation method is as follows:

[0146]

[0147] Where a i This indicates that the action places the currently created virtual machine on the i-th server.

[0148] S333: Calculate servers that do not meet the constraints, further update the action mask, and use the unique constraint identifier apg corresponding to the requesting server. t Corresponding to the type of constraint, constraint categories can be divided into three main categories: no constraint, affinity constraint, and non-affinity constraint. Specific steps are shown in the appendix. Figure 3 As shown, for types without constraints, the condition calculated by S332 is sufficient, and this step can be skipped. For affinity constraints, if there is no corresponding affinity region before, the condition calculated by S332 is sufficient, and this step can be skipped. If there is an affinity region before, such as an affinity rack, then all actions corresponding to servers not in that rack are set to 0. For non-affinity constraints, if there is no corresponding non-affinity region before, the condition calculated by S332 is sufficient, and this step can be skipped. If there is, then all actions corresponding to servers in that region are set to 0.

[0149] S334: Multiply the previously obtained action probability distribution with the action mask to filter out rules that do not meet the requirements, and obtain the final probability distribution.

[0150] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0151] like Figure 5 As shown, the present invention also proposes a computing resource allocation device based on deep reinforcement learning, comprising:

[0152] The initial module acquires a cloud computing resource pool including multiple servers and divides the cloud computing resource pool into: network domain, deployment unit, rack and server. The network domain contains multiple deployment units, which are independent units in the network domain. Each deployment unit contains multiple racks, and each rack is connected to the core network of the deployment unit by one or more network switches. Each rack contains multiple servers.

[0153] The data acquisition module collects the current status information of the cloud computing resource pool, including: hardware performance indicators of each server, status information of virtual machines running on the server, constraint information of the cloud computing resource pool, and request information of computing resources required by each virtual machine.

[0154] The analysis module inputs the feature representation of the state information into the reinforcement learning agent, and analyzes the state of each server by loading a pre-trained deep neural network to obtain the probability distribution matrix of each server being selected by the virtual machine in the request information.

[0155] The allocation module, based on the hardware performance information and the constraint information, determines the compliance of the action corresponding to each element in the probability distribution matrix. Compliant actions are set to a positive number, and non-compliant actions are set to 0, resulting in an action matrix with the same dimension as the probability distribution matrix. The action matrix is ​​multiplied by the probability distribution matrix to obtain the final action probability matrix. Based on the final action probability matrix, the module selects the action corresponding to the largest element in the final action probability matrix for each virtual machine as the current computing resource allocation strategy, and allocates servers to the virtual machines according to the computing resource allocation strategy.

[0156] The aforementioned deep reinforcement learning-based computing resource allocation device, wherein the acquisition module includes:

[0157] Collect actual server operating information, including the server's remaining computing resources; obtain virtual machine status information, including: virtual machine identifier, virtual machine associated constraint identifier, and virtual machine placement location; obtain pre-set constraint information and request information, including affinity type and non-affinity type of constraint information to ensure that the relationship between virtual machines is considered during resource allocation; request information includes the number of requested virtual machines, the amount of computing resources required by each virtual machine, and the constraint identifiers to be followed.

[0158] This analysis module includes:

[0159] The reinforcement learning agent uses a fully connected network structure. The virtual machine state information and the constraint information are represented by fixed-length arrays. Uncreated virtual machines and constraints in the arrays are filled with specified characters. All state information is merged and concatenated. The concatenated state information is input into an encoder composed of multi-layer neural networks. The encoder performs dimensionality reduction, redundant information removal and feature extraction on the input state information through nonlinear transformation to generate a feature representation of the state information.

[0160] Based on this feature representation, the reinforcement learning agent assigns a probability value to each possible operation, representing its tendency to perform that operation, until all virtual machines currently requesting the operation have been assigned probability values, thus obtaining the probability distribution matrix.

[0161] The allocation module includes: judging the action corresponding to each element in the probability distribution matrix, judging the action corresponding to the server whose remaining computing resources are less than the computing resources required by the virtual machine as non-compliant, and judging the action corresponding to the server that does not meet the virtual machine constraints as non-compliant.

[0162] The computing resource allocation device based on deep reinforcement learning includes the following: creating a new initial matrix with the same dimension as the probability distribution matrix; performing compliance judgment on the action corresponding to each element in the probability distribution matrix; setting the value in the initial matrix corresponding to the action of the server with insufficient CPU or memory resources to 0; setting the value in the initial matrix corresponding to the action of the server that does not meet the virtual machine constraints to 0; setting the value in the other initial matrices to 1; and saving the current initial matrix as the action matrix after all elements in the initial matrix have been set.

[0163] The request information includes: creating constraints, and / or creating virtual machines in bulk, and / or deleting virtual machines;

[0164] The creation constraint includes the identification number and category of the newly created constraint;

[0165] This batch creation of virtual machines includes the type and quantity of virtual machines to be created, as well as the constraints that must be met;

[0166] This option deletes a virtual machine, including the number of the specified virtual machine to be deleted.

[0167] like Figure 6 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, including the aforementioned computing resource allocation device based on deep reinforcement learning.

[0168] like Figure 7 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through wired or wireless information transmission schemes. The data acquisition device C is used to collect the node interconnection graph data of the cloud computing resource pool and the current status information of the cloud computing resource pool. The information display device D is used to display the computing resource allocation strategy obtained by the present invention.

[0169] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0170] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the deep reinforcement learning-based computing resource allocation method provided by the above methods.

[0171] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes the aforementioned deep reinforcement learning-based computational resource allocation method. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0172] Figure 8 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0173] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0174] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0175] The computing unit I can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0176] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A computational resource allocation method based on deep reinforcement learning, characterized in that, include: The initial step is to obtain a cloud computing resource pool that includes multiple servers, and divide the cloud computing resource pool into: network domain, deployment unit, rack and server. The network domain contains multiple deployment units, and the deployment unit is an independent unit in the network domain. The deployment unit contains multiple racks, and each rack is connected to the core network of the deployment unit by one or more network switches, and the rack contains multiple servers. The data collection steps involve collecting the current status information of the cloud computing resource pool, including: hardware performance indicators of each server, status information of virtual machines running on the servers, constraint information of the cloud computing resource pool in operation, and request information of computing resources required by each virtual machine. The analysis steps involve inputting the feature representation of the state information into the reinforcement learning agent, and analyzing the state of each server by loading a pre-trained deep neural network to obtain the probability distribution matrix of each server being selected by the virtual machine in the request information. The allocation step involves determining the compliance of each action corresponding to an element in the probability distribution matrix based on the hardware performance metric information and the constraint information. Compliant actions are set to a positive number, while non-compliant actions are set to 0, resulting in an action matrix with the same dimension as the probability distribution matrix. This action matrix is ​​then multiplied by the probability distribution matrix to obtain the final action probability matrix. Based on this final action probability matrix, the action corresponding to the largest element in the final action probability matrix is ​​selected for each virtual machine as the current computing resource allocation strategy. A server is then allocated to the virtual machine according to this computing resource allocation strategy.

2. The computational resource allocation method based on deep reinforcement learning as described in claim 1, characterized in that, The data collection process includes: Collect actual server operating information, including the server's remaining computing resources; obtain virtual machine status information, including: virtual machine identifier, virtual machine associated constraint identifier, and virtual machine placement location; obtain pre-set constraint information and request information, including affinity type and non-affinity type of constraint information to ensure that the relationship between virtual machines is considered during resource allocation; request information includes the number of requested virtual machines, the amount of computing resources required by each virtual machine, and the constraint identifiers to be followed. The analytical steps include: The reinforcement learning agent uses a fully connected network structure. The virtual machine state information and the constraint information are represented by fixed-length arrays. Uncreated virtual machines and constraints in the arrays are filled with specified characters. All state information is merged and concatenated. The concatenated state information is input into an encoder composed of multi-layer neural networks. The encoder performs dimensionality reduction, redundant information removal and feature extraction on the input state information through nonlinear transformation to generate a feature representation of the state information. Based on this feature representation, the reinforcement learning agent assigns a probability value to each possible operation, representing its tendency to perform that operation, until all virtual machines currently requesting the operation have been assigned probability values, thus obtaining the probability distribution matrix. The allocation step includes: judging the action corresponding to each element in the probability distribution matrix, judging the actions of servers with remaining computing resources less than the computing resources required by the virtual machine as non-compliant, and judging the actions of servers that do not meet the virtual machine constraints as non-compliant.

3. The computational resource allocation method based on deep reinforcement learning as described in claim 1 or 2, characterized in that, The allocation steps include: creating a new initial matrix with the same dimension as the probability distribution matrix; performing compliance checks on the actions corresponding to each element in the probability distribution matrix; setting the values ​​in the initial matrix corresponding to actions of servers with insufficient CPU or memory resources to 0; setting the values ​​in the initial matrix corresponding to actions of servers that do not meet virtual machine constraints to 0; setting the values ​​in the other initial matrices to 1; and saving the current initial matrix as the action matrix after all elements in the initial matrix have been set.

4. The computational resource allocation method based on deep reinforcement learning as described in claim 1 or 2, characterized in that, The request information includes: creating constraints, and / or creating virtual machines in bulk, and / or deleting virtual machines; The creation constraint includes the identification number and category of the newly created constraint; This batch creation of virtual machines includes the type and quantity of virtual machines to be created, as well as the constraints that must be met; This option deletes a virtual machine, including the number of the specified virtual machine to be deleted.

5. A computing resource allocation device based on deep reinforcement learning, characterized in that, include: The initial module acquires a cloud computing resource pool including multiple servers and divides the cloud computing resource pool into: network domain, deployment unit, rack and server. The network domain contains multiple deployment units, which are independent units in the network domain. Each deployment unit contains multiple racks, and each rack is connected to the core network of the deployment unit by one or more network switches. Each rack contains multiple servers. The data acquisition module collects the current status information of the cloud computing resource pool, including: hardware performance indicators of each server, status information of virtual machines running on the server, constraint information of the cloud computing resource pool, and request information of computing resources required by each virtual machine. The analysis module inputs the feature representation of the state information into the reinforcement learning agent, and analyzes the state of each server by loading a pre-trained deep neural network to obtain the probability distribution matrix of each server being selected by the virtual machine in the request information. The allocation module, based on the hardware performance information and the constraint information, determines the compliance of the action corresponding to each element in the probability distribution matrix. Compliant actions are set to a positive number, and non-compliant actions are set to 0, resulting in an action matrix with the same dimension as the probability distribution matrix. The action matrix is ​​multiplied by the probability distribution matrix to obtain the final action probability matrix. Based on the final action probability matrix, the module selects the action corresponding to the largest element in the final action probability matrix for each virtual machine as the current computing resource allocation strategy, and allocates servers to the virtual machines according to the computing resource allocation strategy.

6. The computing resource allocation device based on deep reinforcement learning as described in claim 5, characterized in that, The acquisition module includes: Collect actual server operating information, including the server's remaining computing resources; obtain virtual machine status information, including: virtual machine identifier, virtual machine associated constraint identifier, and virtual machine placement location; obtain pre-set constraint information and request information, including affinity type and non-affinity type of constraint information to ensure that the relationship between virtual machines is considered during resource allocation; request information includes the number of requested virtual machines, the amount of computing resources required by each virtual machine, and the constraint identifiers to be followed. This analysis module includes: The reinforcement learning agent uses a fully connected network structure. The virtual machine state information and the constraint information are represented by fixed-length arrays. Uncreated virtual machines and constraints in the arrays are filled with specified characters. All state information is merged and concatenated. The concatenated state information is input into an encoder composed of multi-layer neural networks. The encoder performs dimensionality reduction, redundant information removal and feature extraction on the input state information through nonlinear transformation to generate a feature representation of the state information. Based on this feature representation, the reinforcement learning agent assigns a probability value to each possible operation, representing its tendency to perform that operation, until all virtual machines currently requesting the operation have been assigned probability values, thus obtaining the probability distribution matrix. The allocation module includes: judging the action corresponding to each element in the probability distribution matrix, judging the action corresponding to the server whose remaining computing resources are less than the computing resources required by the virtual machine as non-compliant, and judging the action corresponding to the server that does not meet the virtual machine constraints as non-compliant.

7. The computing resource allocation device based on deep reinforcement learning as described in claim 5 or 6, characterized in that, The allocation module includes: creating a new initial matrix with the same dimension as the probability distribution matrix; performing compliance judgment on the action corresponding to each element in the probability distribution matrix; setting the value in the initial matrix corresponding to the action of the server with insufficient CPU or memory resources to 0; setting the value in the initial matrix corresponding to the action of the server that does not meet the virtual machine constraints to 0; setting the value in the other initial matrices to 1; and saving the current initial matrix as the action matrix after all elements in the initial matrix have been set. The request information includes: creating constraints, and / or creating virtual machines in bulk, and / or deleting virtual machines; The creation constraint includes the identification number and category of the newly created constraint; This batch creation of virtual machines includes the type and quantity of virtual machines to be created, as well as the constraints that must be met; This option deletes a virtual machine, including the number of the specified virtual machine to be deleted.

8. An electronic device, characterized in that, The device includes a deep reinforcement learning-based computing resource allocation apparatus as described in any one of claims 5-7, wherein the electronic device is connected to an information display device, which displays the computing resource allocation strategy using user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep reinforcement learning-based computational resource allocation method according to any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the deep reinforcement learning-based computational resource allocation method as described in any one of claims 1-4.

Images