A computing power network scheduling method, device and medium based on deep reinforcement learning

Abstract

The present invention discloses a computing power network scheduling method, device and medium based on deep reinforcement learning, the method comprising: obtaining computing power network environment information; determining target scheduling model information based on computing power network environment information and initial scheduling model information; target scheduling model information including M target resource scheduling models; target resource scheduling model including a first target resource scheduling model and / or a second target resource scheduling model; determining target computing power network resource scheduling result information based on target scheduling model information and computing power network environment information; target computing power network resource scheduling result information including computing power scheduling type and computing power parameter information. The computing power network scheduling method of the present invention can schedule computing power network resources based on deep reinforcement learning, which can improve scheduling efficiency and accuracy, thereby improving user experience.

Description

Technical Field

[0001] The present invention relates to the field of scheduling technology, and in particular to a computing power network scheduling method, device and medium based on deep reinforcement learning. Background Art

[0002] The current computing resource allocation and management is mainly achieved through advanced virtualization technologies, including cloud native architecture and cloud computing services. These technologies allow flexible scheduling and management of resources through platforms such as OpenStack or Kubernetes. During the resource scheduling process, the system will give priority to the availability of computing nodes, that is, their idle state, as the main reference standard for resource allocation. In addition, the existing computing scheduling technology will customize the allocation strategy according to the specific needs and characteristics of the task. For example, for tasks that require processing large amounts of data and performing complex calculations, the system will adopt a more extensive resource allocation method. This method can effectively disperse the computing load and avoid excessive computing pressure on a single data center, thereby ensuring the stability and efficiency of the entire computing network. Through this intelligent resource scheduling strategy, not only can the utilization rate of computing resources be improved, but also the resource allocation can be flexibly adjusted according to the needs of different tasks to meet the needs of different users and application scenarios. This flexible and intelligent resource management method is an important feature of the modern cloud computing and big data era, and it is also one of the key technologies to promote the development of computing networks.

[0003] With the vigorous development of artificial intelligence and mobile Internet technology, the transmission and analysis of massive data have led to a decentralized trend of computing resources in the network, and the resource utilization has shown problems of poor load balancing and insufficient utilization. At present, most of them obtain the optimal path of computing tasks by weighted calculation of network and computing power resource state parameters, lacking a load balancing mechanism that comprehensively considers network and computing power resources, that is, the current allocation and scheduling mechanism may lead to uneven distribution of loads among different nodes and low resource utilization efficiency. This situation not only causes a waste of computing power resources, but also makes it difficult to effectively manage and control computing power resources. Ultimately, this unbalanced resource allocation and inefficient resource utilization may have a negative impact on the reliability and stability of applications. Therefore, it is necessary to provide a computing power network scheduling method and device based on deep reinforcement learning, which can schedule computing power network resources based on deep reinforcement learning, improve the efficiency and accuracy of scheduling, and thus improve the user experience. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a computing power network scheduling method, device and medium based on deep reinforcement learning, so as to improve the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0005] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a computing power network scheduling method based on deep reinforcement learning, and the method includes:

[0006] Obtain computing power network environment information;

[0007] Based on the computing power network environment information and the initial scheduling model information, target scheduling model information is determined; the target scheduling model information includes M target resource scheduling models; the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model;

[0008] Based on the target scheduling model information and the computing power network environment information, the target computing power network resource scheduling result information is determined; the target computing power network resource scheduling result information includes the computing power scheduling type and computing power parameter information.

[0009] A second aspect of an embodiment of the present invention discloses a computing power network scheduling device based on deep reinforcement learning, the device comprising:

[0010] The acquisition module is used to obtain computing power network environment information;

[0011] A first determination module is used to determine target scheduling model information based on computing power network environment information and initial scheduling model information; the target scheduling model information includes M target resource scheduling models, and the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model;

[0012] The second determination module is used to determine the target computing power network resource scheduling result information based on the target scheduling model information and the computing power network environment information; the target computing power network resource scheduling result information includes the computing power scheduling type and computing power parameter information.

[0013] The third aspect of the present invention discloses another computing power network scheduling device based on deep reinforcement learning, the device comprising:

[0014] A memory storing executable program code;

[0015] a processor coupled to the memory;

[0016] The processor calls the executable program code stored in the memory to execute part or all of the steps in the computing power network scheduling method disclosed in the first aspect of the embodiment of the present invention.

[0017] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps in the computing power network scheduling method disclosed in the first aspect of an embodiment of the present invention.

[0018] The above scheme of the present invention has the following beneficial effects:

[0019] In an embodiment of the present invention, by obtaining computing power network environment information, then based on the computing power network environment information and the initial scheduling model information, the target scheduling model information is determined, and finally based on the target scheduling model information and the computing power network environment information, the target computing power network resource scheduling result information is determined. When determining the target computing power network resource scheduling result, the computing power network environment information and the resource scheduling model information are taken into account, which effectively improves the scheduling efficiency and accuracy of the computing power network resources, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of a scenario of a computing power network scheduling system provided by an embodiment of the present invention;

[0022] Figure 2 It is a flowchart of a computing power network scheduling method based on deep reinforcement learning disclosed in an embodiment of the present invention;

[0023] Figure 3 It is a structural schematic diagram of a computing power network scheduling device based on deep reinforcement learning disclosed in an embodiment of the present invention;

[0024] Figure 4 It is a structural diagram of another computing power network scheduling device based on deep reinforcement learning disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0027] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] In the present invention, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in the present invention is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0029] It should be noted that since the method of the embodiment of the present invention is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0030] The embodiments of the present invention provide a computing power network scheduling method, device and medium based on deep reinforcement learning, which are described in detail below.

[0031] See also Figure 1 , Figure 1 This is a scenario diagram of a computing power network scheduling system provided in an embodiment of the present invention. The computing power network scheduling system may include a computer device 100, in which a computing power network scheduling apparatus is integrated.

[0032] In the embodiment of the present invention, the computer device 100 is mainly used to obtain computing power network environment information;

[0033] Based on the computing power network environment information and the initial scheduling model information, target scheduling model information is determined; the target scheduling model information includes M target resource scheduling models, and the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model;

[0034] Based on the target scheduling model information and the computing power network environment information, the target computing power network resource scheduling result information is determined; the target computing power network resource scheduling result information includes the computing power scheduling type and computing power parameter information.

[0035] It can improve the scheduling efficiency and accuracy of computing network resources, thereby improving user experience.

[0036] In the embodiment of the present invention, the computer device 100 may be an independent server, or a server network or server cluster composed of servers. For example, the computer device 100 described in the embodiment of the present invention includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0037] It is understandable that the computer device 100 used in the embodiment of the present invention may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific computer device 100 may be a desktop terminal or a mobile terminal, and the computer device 100 may also be one of a mobile phone, a tablet computer, a laptop computer, etc.

[0038] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only an example of an application scenario of the solution of the present invention and does not constitute a limitation on the application scenario of the solution of the present invention. Other application environments may also include Figure 1 More or less computer equipment as shown in Figure 1 Only one computer device is shown. It can be understood that the computing power network scheduling system can also include one or more other services, which are not specifically limited here.

[0039] In addition, if Figure 1 As shown, the computing power network scheduling system may also include a memory 200 for storing data, such as image data, location information, etc.

[0040] It should be noted that Figure 1 The scenario diagram of the computing power network scheduling system shown is only an example. The computing power network scheduling system and scenario described in the embodiment of the present invention are intended to more clearly illustrate the technical solution of the embodiment of the present invention, and do not constitute a limitation on the technical solution provided by the embodiment of the present invention. Ordinary technicians in this field can know that with the evolution of the computing power network scheduling system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0041] The present invention discloses a computing network scheduling method and device based on deep reinforcement learning, which is beneficial to improving the efficiency and accuracy of computing network resource scheduling, thereby improving the user experience. The following are detailed descriptions.

[0042] Embodiment 1

[0043] See also Figure 2 , Figure 2 It is a flowchart of a computing power network scheduling method based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 2 The computing power network scheduling method described is applied to a management system, such as a local server or a cloud server for management, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the computing power network scheduling method may include the following operations:

[0044] 101. Obtain computing power network environment information.

[0045] 102. Based on the computing power network environment information and the initial scheduling model information, determine the target scheduling model information.

[0046] In the embodiment of the present invention, the target scheduling model information includes M target resource scheduling models; the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model.

[0047] It should be noted that the above M is a positive integer not less than 2, and the embodiment of the present invention is not limited thereto.

[0048] 103. Based on the target scheduling model information and the computing power network environment information, determine the target computing power network resource scheduling result information.

[0049] In the embodiment of the present invention, the target computing power network resource scheduling result information includes computing power scheduling type and computing power parameter information. The computing power scheduling type is used to describe the type of task scheduling, such as not scheduling to other service nodes, scheduling to other service nodes, etc. The computing power parameter information is used to describe the remaining computing power resources, computing power deployment status, etc. The computing power network environment information includes environment attribute information, that is, the computing power type in the current computing power network environment, the computing power priority under the same computing power type, and the computing power available remaining resource information.

[0050] It should be noted that the above-mentioned target computing power network resource scheduling result information is used to indicate the scheduling of computing power resources of the computing power network to balance the workload and improve the utilization efficiency of computing power resources, and the embodiments of the present invention are not limited thereto.

[0051] It should be noted that, based on extensive network connections, the computing power network effectively manages unused computing, storage and network resources within the system to meet the needs of upper-level businesses for computing power services. This technology is expected to achieve on-demand task scheduling and flexible resource sharing, showing great potential. However, with the expansion of network scale and the increase in complexity, large-scale collaboration between multiple computing power service nodes across operators or domains has become extremely challenging due to problems such as unbalanced workloads and limited resources of a single service node. How to effectively manage and utilize these distributed ubiquitous computing powers is a realistic problem that needs to be solved urgently. The computing power network scheduling method of the present invention uses a scheduling task tree to convert the computing power resource scheduling rules into an efficiently executable scheduling program logic, solve the problem of standardized expression of the computing power resource scheduling process, and on this basis, integrate task nodes based on reinforcement learning, and use deep learning algorithms (such as RNN, CNN, etc.) to represent the computing power network environment, thereby improving the efficient and flexible feasibility of computing power resources.

[0052] In the related art, the environmental attribute information includes the corresponding attribute number and attribute priority. The attribute number of the environmental attribute information can be set by the user according to actual needs, or it can be numbered by the system according to time or regional information according to preset rules. Its representation form can be A001, B001, or pure Arabic numerals, or pure English letters. It is mainly used for identity identification and verification, and the embodiment of the present invention is not limited. The attribute priority can be set by the user according to actual needs, or it can be set from high to low by the system according to the time sequence of generating information, and the embodiment of the present invention is not limited. The model priority can be set according to actual needs (that is, generated in response to the user's selection instruction through human-computer interaction at the time of initialization), or the system determines it according to historical data. The model priority and the attribute priority are in the same form. For example, if the model priority is selected to be represented by a number from 1 to 5, and the larger the number, the higher the priority, then the attribute priority is also represented by a number from 1 to 5, and the larger the number, the higher the priority. The initial scheduling model information includes the task number of the task node corresponding to the initial scheduling model, and each task number corresponds to at least one initial scheduling model. The task node is the node corresponding to the scheduling task in the scheduling task tree. The initial scheduling model information also includes multiple initial scheduling models. The initial scheduling model is one of a first initial scheduling model and a second initial scheduling model. The first initial scheduling model corresponds to the first target resource scheduling model, and the second initial scheduling model corresponds to the second target resource scheduling model.

[0053] It should be noted that in reinforcement learning related technologies, expert experience can be used to replace random exploration strategies, and this application explains this. When reinforcement learning performs data sampling and the strategy outputs actions, it generally adopts an exploration strategy, especially in the early stages of training, and generally prefers an exploration strategy. Taking the epsilon decay greedy strategy as an example, the formula for its decision-making action is as follows:

[0054]

[0055] in represents the parameter in the epsilon decay greedy strategy, which is generally a value between 0 and 1. rand(0,1) represents a randomly generated value between 0 and 1. s represents the current state information of the input. θ(s) represents the action output by the reinforcement learning training strategy. rand(s) represents the action output by the random strategy. π(a│s) represents the final output action under the s state. In the early stage, a larger value is generally given. , making the strategy more inclined to random strategy, making more possible explorations, and then gradually decaying with the increase in the number of training times , until it becomes a smaller threshold. When using expert experience to replace the random strategy, two other parameters can be introduced and β to control the weights of the expert experience strategy, random strategy, and model learning strategy. The formula is as follows:

[0056]

[0057] in represents the expert experience strategy, To control the trade-off factor between the reinforcement learning strategy and the exploration strategy, β is used to regulate the weights of the expert strategy and the random strategy, because in the later stage of exploration, more random exploration is needed to better find the optimal solution. The and β parameters will continue to decay with the increase of training times, and the decay rate of β will be greater than This is a mode of introducing expert experience for reinforcement learning models with discrete actions or greedy strategies as exploration strategies. However, there is also a type of reinforcement learning model based on strategies, which mainly uses Boltzman distribution as its exploration strategy. For this type of reinforcement learning model, the way to apply expert experience is similar to the above, but Boltzman distribution as its exploration strategy is based on probability to select actions, without random strategies. At this time, the parameter is introduced. , then add the strategy probability output by reinforcement learning and the strategy probability output by expert experience, and then select a new action. The probability formula for selecting an action is:

[0058]

[0059] in represents the probability of the final action selection, represents the probability of the expert action output, Represents the action probability output by the reinforcement learning strategy, through the parameter To adjust the weight, as the number of training times increases, the parameters It keeps decaying until it reaches 0. When performing attenuation, you can use the estimated piecewise function to attenuate, and try to keep using expert experience in the first 1 / 3 of the training rounds. Expert experience will not be used in the future. Therefore, if the total number of training rounds is , the attenuation amplitude is , generally take =0.9, n represents the current number of training rounds, then the formula is:

[0060]

[0061] Try to divide reinforcement learning exploration into three periods: expert experience-based, random strategy-based, and reinforcement learning algorithm-based.

[0062] It should be noted that the target scheduling model information is determined based on the computing power network environment information and the initial scheduling model information. This is determined by the consistency between the attribute number of the environmental attribute information corresponding to the computing power network environment information and the task number corresponding to the task node of the scheduling task tree corresponding to the initial scheduling model information. That is, the initial scheduling model corresponding to the task number consistent with the attribute number is determined as the target resource scheduling model, thereby improving the efficiency and accuracy of model selection through number consistency, thereby improving the scheduling efficiency of computing power resources, and minimizing the minimum average operation completion time of task loading using computing power resources. This is not limited to the embodiments of the present invention.

[0063] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0064] In an optional embodiment, the target computing power network resource scheduling result information is determined based on the target scheduling model information and the computing power network environment information, including:

[0065] Determine whether there is a first target resource scheduling model among all target resource scheduling models in the target scheduling model information, and obtain an existence determination result;

[0066] When the judgment result is yes, it is judged whether the attribute priority corresponding to the environment attribute information in the computing power network environment information is consistent with the model priority of the first target resource scheduling model to obtain a consistent judgment result;

[0067] When the consistent judgment result is yes, the computing power network environment information is processed based on the first target resource scheduling model to obtain the target computing power network resource scheduling result information;

[0068] When the consistent judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information;

[0069] When the judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information.

[0070] It should be noted that by judging whether the attribute priority corresponding to the environmental attribute information is consistent with the model priority of the first target resource scheduling model, the target resource scheduling model can be quickly determined when there are multiple initial scheduling models in the same task node, so as to efficiently determine the scheduling rules for computing resource scheduling according to the task attributes corresponding to the current target scheduling task, thereby improving the efficiency and accuracy of resource scheduling in the computing network. This is not limited in the embodiments of the present invention. Furthermore, the above-mentioned priority can be determined based on a preset number, the larger the number, the higher the priority, or it can be determined based on 26 English letters such as A / B / C, the earlier the alphabetical order, the higher the priority, which is not limited in the embodiments of the present invention.

[0071] Since multiple first target resource scheduling models and multiple second target resource scheduling models are determined from all initial scheduling models based on the attribute number and the task number when determining the target scheduling model information, both the first target resource scheduling model and the second target resource scheduling model may exist, or only the first target resource scheduling model or the second target resource scheduling model may exist. The difference between the first target resource scheduling model and the second target resource scheduling model lies in the different scales and computing power, and the scale and computing power of the first target resource scheduling model are smaller than those of the second target resource scheduling model.

[0072] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0073] In another optional embodiment, the computing power network environment information is processed based on the first target resource scheduling model to obtain target computing power network resource scheduling result information, including:

[0074] The computing power network environment information is converted and processed to obtain the computing power network space state information; the computing power network space state information represents the data information of the multi-dimensional tensor structure;

[0075] The computing power network space status information is processed by the first resource scheduling model in the first target resource scheduling model to obtain the target computing power network resource scheduling result information; the first resource scheduling model includes a first sub-resource scheduling model, a second sub-resource scheduling model and a third sub-resource scheduling model; the second sub-resource scheduling model and the third sub-resource scheduling model are connected in parallel to the first sub-resource scheduling model.

[0076] It should be noted that the above-mentioned conversion processing of the computing power network environment information is a matrix conversion of the computing power network environment information, so as to form a standard structured computing power network space status information, so that the first resource scheduling model can perform accurate and efficient data processing, and the embodiments of the present invention are not limited thereto.

[0077] It should be noted that the above-mentioned second target resource scheduling model can be a network model based on a combination of deep neural networks (DNNs) and policy gradient methods, wherein the deep neural network (DNNs) fits the computing power resource data, and then optimizes the resource allocation strategy through the policy gradient method, thereby obtaining the target computing power network resource scheduling result information. Furthermore, the second target resource scheduling model can also be a multi-agent model constructed based on the independent Q learning (IQL, ImplicitQ-learning) algorithm, which can adopt multi-agent reinforcement learning to allow each agent to learn to optimize resource allocation in a local environment while considering the global optimization goal, which is not limited in the embodiments of the present invention.

[0078] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0079] In another optional embodiment, the computing power network space state information is processed using the first resource scheduling model in the first target resource scheduling model to obtain target computing power network resource scheduling result information, including:

[0080] The computing power network space state information is convolved N times using the first sub-resource scheduling model to obtain first state feature information; N is a positive integer not less than 2; the N convolutions are implemented based on the convolution layer in the first sub-resource scheduling model;

[0081] Using the first sub-resource scheduling model, a spatial pyramid pooling operation is performed on the first state feature information to obtain second state feature information; the spatial pyramid pooling operation is implemented based on the pooling layer in the first sub-resource scheduling model;

[0082] The second state characteristic information is fully connected by using the first sub-resource scheduling model to obtain third state characteristic information; the full connection processing is implemented based on the full connection layer in the first sub-resource scheduling model;

[0083] The third state feature information is convolved using the second sub-resource scheduling model to obtain computing power parameter information;

[0084] The third state characteristic information is processed using the third sub-resource scheduling model to obtain the computing power scheduling type.

[0085] It should be noted that the convolution layer in the above-mentioned first sub-resource scheduling model is composed of a sub-convolution layer with a convolution kernel size of 5×5 and a sub-convolution layer with a convolution kernel size of 3×3 connected in series, which is not limited in the embodiment of the present invention. Furthermore, the step size of the sub-convolution layer is 1, which is not limited in the embodiment of the present invention. The above-mentioned use of the first resource scheduling model to obtain computing power parameter information and computing power scheduling type can be understood as inputting the computing power network space state information into the first sub-resource scheduling model in the first resource scheduling model, the first sub-resource scheduling model outputs the third state feature information, and the third state feature information is respectively input into the second sub-resource scheduling model and the third sub-resource scheduling model, the second sub-resource scheduling model outputs computing power parameter information, and the third sub-resource scheduling model outputs the computing power scheduling type.

[0086] It should be noted that the data format of the third state characteristic information is 128×128×32, which is not limited in the embodiment of the present invention.

[0087] It should be noted that the above-mentioned use of the first sub-resource scheduling model to perform spatial pyramid pooling on the first state feature information is to solve the problem of input size limitation of the network, so as to use pooling of different scales to obtain pooling features of the same size (that is, using pooling layers of different sizes to extract features of different scales to obtain multi-scale features). Furthermore, the use of the first sub-resource scheduling model to perform full connection processing on the second state feature information can connect features of different scales to achieve better feature expression, which is not limited in the embodiment of the present invention.

[0088] It should be noted that the second sub-resource scheduling model is a convolutional network, whose convolution kernel size is 1×1 and whose step size is 1, which is not limited in the embodiment of the present invention. Further, the data specification of the computing power parameter information is 128×128×1, which is not limited in the embodiment of the present invention.

[0089] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0090] In yet another optional embodiment, the third state characteristic information is processed using the third sub-resource scheduling model to obtain a computing power scheduling type, including:

[0091] The third state feature information is converted for the first time using the third sub-resource scheduling model to obtain feature vector information; the feature vector information represents data information with a dimension of 1×256; the first conversion processing of the third state feature information is implemented based on a fully connected neural network in the third sub-resource scheduling model;

[0092] The feature vector information is converted for the second time using the third sub-resource scheduling model to obtain computing power resource scheduling distribution information; the resource scheduling distribution value in the computing power resource scheduling distribution information represents the distribution of computing power resources in the computing power network; the feature vector information is converted for the second time based on the distribution conversion model in the third sub-resource scheduling model;

[0093] Based on the resource scheduling relationship table, the computing power resource scheduling probability information is matched and screened to obtain the computing power scheduling type; the resource scheduling relationship table represents the corresponding relationship between the computing power scheduling type corresponding to the resource scheduling distribution value and the resource scheduling distribution threshold.

[0094] It can be understood that the structure of the third sub-resource scheduling model is a fully connected neural network and a distributed conversion model connected in sequence.

[0095] It should be noted that the number of resource scheduling distribution values ​​in the above computing power resource scheduling distribution information is between 3-10, so as to ensure that the computing power resources of different network nodes in the computing power network can be fully utilized when computing power resources are scheduled, and the computing power resources can be quickly scheduled, thereby improving the effective utilization rate of computing power resources. The embodiments of the present invention are not limited to this.

[0096] It should be noted that the above resource scheduling distribution value is a numerical value between 0 and 1, which is not limited in the embodiment of the present invention.

[0097] In this optional embodiment, as an optional implementation mode, the above-mentioned matching and screening processing of the computing power resource scheduling probability information based on the resource scheduling relationship table to obtain the computing power scheduling type includes:

[0098] Sort all resource scheduling distribution values ​​in the computing power resource scheduling probability information from large to small to obtain a resource scheduling distribution value sequence;

[0099] Based on the resource scheduling relationship table, the resource scheduling distribution values ​​in the resource scheduling distribution value sequence are compared and analyzed in sequence to obtain a comparison analysis result sequence corresponding to the resource scheduling distribution value; the comparison analysis results in the comparison analysis result sequence include computing power resources that can be scheduled and computing power resources that cannot be scheduled;

[0100] Selecting a comparative analysis result from the comparative analysis result sequence in sequence as a to-be-processed comparative analysis result, and when the to-be-selected comparative analysis result is a schedulable computing resource, determining the to-be-processed comparative analysis result as a target comparative analysis result;

[0101] Determine that the scheduling type corresponding to the schedulable computing resources is the computing power scheduling type.

[0102] It should be noted that the above resource scheduling relationship table sorts the available computing resources according to the order of the network nodes in the computing network, forming a relationship pair of network node-available computing resources-computing resource ratio (computing resource ratio represents the proportion of available computing resources to all available computing resources, which is not limited in the embodiment of the present invention). Further, the schedulable computing resources represent that the resource scheduling distribution value is not less than the schedulable computing resources, which is not limited in the embodiment of the present invention.

[0103] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0104] In an optional embodiment, after processing the computing power network space state information using the first resource scheduling model in the first target resource scheduling model to obtain the target computing power network resource scheduling result information, the method further includes:

[0105] Obtain computing power resource usage feedback information corresponding to the target computing power network resource scheduling result information;

[0106] Calculate the computing power resource usage feedback information based on the computing power usage evaluation formula to obtain the usage evaluation value;

[0107] The calculation power usage evaluation formula is:

[0108] ;

[0109] In the formula, To use the evaluation value, , and They are the computing power resources in the first computing power parameter information, the computing power resources in the second computing power parameter information, and the computing power resources in the The number of computing resources in the computing power parameter information, , , , and They are the first evaluation coefficient, the second evaluation coefficient, the first weight coefficient, the second weight coefficient and the Weight coefficient, For the Weight coefficient, To use the first computing power parameter information The estimated value of computing resources, To use the second computing power parameter information The estimated value of computing resources, To use The computing power parameter information The estimated value of computing resources. To use the first computing power parameter information The estimated value of computing resources. To use the second computing power parameter information The estimated value of computing resources. To use The computing power parameter information The estimated value of a computing resource;

[0110] Using the second resource scheduling model in the first target resource scheduling model to perform error calculation processing on the usage evaluation value to obtain a usage error evaluation value;

[0111] The first resource scheduling model is updated using the usage error evaluation value.

[0112] It should be noted that the first weight coefficient, the second weight coefficient and the The weight coefficient can be set by the user or a default value given by the system, which is not limited in the embodiment of the present invention. Furthermore, its value is between 0-1, which is not limited in the embodiment of the present invention. Furthermore, the first weight coefficient is not less than 0.6, and the weight coefficients after the second weight coefficient are not greater than 0.5, so as to ensure that the first computing power parameter information has a greater impact on the overall evaluation, that is, to achieve priority call to the computing power resources corresponding to the first computing power parameter information, which is not limited in the embodiment of the present invention. Furthermore, the above weight coefficients have All weight coefficients are arranged in sequence, from the first to the , further, first, second, ,..., No. The position of the weight coefficient is not limited in this embodiment of the present invention. To use the first computing power parameter information The estimated value of a computing resource; To use the second computing power parameter information The estimated value of a computing resource; To use The computing power parameter information The estimated value of a computing resource.

[0113] It should be noted that the first evaluation coefficient is a positive number between 0.8 and 1, which is not limited in the embodiment of the present invention. The second evaluation coefficient is a positive number not greater than 0.3, which is not limited in the embodiment of the present invention. The computing power resource usage feedback information is used to describe the number and related information of computing power resources in each computing power parameter information in the target computing power network resource scheduling result information.

[0114] It should be noted that the above computing power resources represent the computing power resources corresponding to the scheduled network nodes, which is not limited in the embodiment of the present invention. Further, the computing power resources in the computing power parameter information are sorted by network nodes, which is not limited in the embodiment of the present invention.

[0115] It should be noted that the above evaluation value can be directly given by the system (such as calculating the ratio of the computing power resources actually used to the computing power resources called, which is not limited in the embodiment of the present invention), or it can be evaluated by the user, which is not limited in the embodiment of the present invention.

[0116] It should be noted that the above-mentioned updating of the first resource scheduling model using the usage error evaluation value is to use the usage error evaluation value to perform gradient calculation, and update the parameters of the first resource scheduling model according to the learning rate given by the system, which is not limited in the embodiment of the present invention. Further, the above-mentioned learning rate can be 0.0001, which is not limited in the embodiment of the present invention.

[0117] It should be noted that the above-mentioned second resource scheduling model can be constructed based on the convolutional neural network in the deep learning algorithm, or based on the Actor-Critic model, and it can also be constructed based on historical scheduling data, which is not limited in the embodiment of the present invention.

[0118] It should be noted that when the second resource scheduling model is constructed based on historical scheduling data, the construction method is: detecting whether there is historical scheduling data;

[0119] When historical scheduling data is detected, clone learning is performed on the historical scheduling data to construct a second resource scheduling model;

[0120] When no historical scheduling data is detected, for any scheduling experience data in the scheduling data pool preset by the user, it is determined whether the reward function value corresponding to the scheduling experience data is greater than or equal to the reward threshold preset by the user, and a first determination result is obtained;

[0121] When the reward judgment result is yes, determining the scheduling experience data as a scheduling training data;

[0122] All scheduling training data are cloned to build a second resource scheduling model.

[0123] It should be noted that the above-mentioned clonal learning is to fit and form a model by imitating the resource scheduling actions corresponding to the historical scheduling data. It can be implemented in python based on linear regression, quadratic regression and random sampling consensus (RANSAC) algorithm, or it can be implemented based on MATLAB's simulink module, which is not limited in the embodiments of the present invention.

[0124] It should be noted that the scheduling data pool is constructed based on reinforcement learning, which is not limited in the embodiment of the present invention.

[0125] It should be noted that the reward threshold changes dynamically and is not limited in the embodiment of the present invention.

[0126] It should be noted that the reward function value is obtained by calculating the experience data using a preset reward function, and the embodiment of the present invention does not limit this.

[0127] It should be noted that the reward function is determined based on reinforcement learning, and this embodiment of the present invention does not limit this.

[0128] It should be noted that the above-mentioned historical scheduling data and scheduling data pool represent past computing network resource scheduling records existing in the system, or are copied to the system by the user, and the embodiments of the present invention are not limited thereto.

[0129] It can be understood that the first target resource scheduling model includes a first resource scheduling model and a second resource scheduling model, and the second resource scheduling model inputs a usage evaluation value and outputs an error evaluation value for updating the first resource scheduling model.

[0130] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0131] In another optional embodiment, the initial scheduling model information is obtained based on the following method:

[0132] Obtain computing resource scheduling task information;

[0133] Decompose the computing resource scheduling task information to obtain target scheduling task information; the target scheduling task information includes L target scheduling tasks; L is a positive integer not less than 2;

[0134] A scheduling task tree is constructed based on the task relationship of each target scheduling task in the target scheduling task information; the scheduling task tree includes a plurality of task nodes; each task node corresponds to a target scheduling task;

[0135] Based on the basic scheduling model information, each task node in the scheduling task tree is subjected to model adjustment processing to obtain initial scheduling model information; the initial scheduling model information includes several initial scheduling models; each task node corresponds to K initial scheduling models; K is a positive integer not less than 1.

[0136] It should be noted that the edge between two task nodes in the scheduling task tree is the task relationship between the corresponding two target scheduling tasks. The above task relationship represents the execution order relationship of the target scheduling tasks. For example, task A and task B can be executed in parallel, task C is executed after task A, and task D is executed after task B. The embodiment of the present invention is not limited to this.

[0137] It should be noted that the above-mentioned initial scheduling model includes a first initial scheduling model and / or a second initial scheduling model. The task node corresponds to at least one initial scheduling model. The task node has its own task number, so the task number corresponds to at least one initial scheduling model, which is not limited in the embodiment of the present invention.

[0138] It should be noted that the above-mentioned method realizes standardized formal expression and process control of computing power resource scheduling tasks by decomposing the computing power resource scheduling tasks in the computing power resource scheduling task information and constructing a scheduling task tree based on the task relationship, that is, formally expressing the logical relationship between the parent and child nodes of the scheduling task tree, standardizing the computing power scheduling process of the computing power resource scheduling tasks, and effectively improving the scheduling efficiency and accuracy of computing power resources, which is not limited to the embodiments of the present invention.

[0139] It should be noted that the above basic scheduling model information includes several basic scheduling models and the task relationship of each target scheduling task, each task node corresponds to at least one basic scheduling model, and the basic scheduling model is one of the first basic scheduling model and the second basic scheduling model. Further, the above first basic resource scheduling model corresponds to the first initial resource scheduling model; the second basic resource scheduling model corresponds to the second initial resource scheduling model. Further, the above-mentioned first basic resource scheduling model and the second basic resource scheduling model in the basic scheduling model information are used to adjust the model of the task node according to user-specific requirements such as parameters and scheduling rules (such as adjusting the model parameters of the first basic resource scheduling model and the second basic resource scheduling model, or adjusting the logical control rules of the first basic resource scheduling model, which can be adjusted by the user, or can be automatically completed by the system based on the replacement and update method of data, which is not limited in the embodiment of the present invention), thereby solving the problem that it is difficult to achieve large-scale collaboration between multiple computing service nodes across operators or domains due to problems such as unbalanced workload and limited resources of a single service node. Furthermore, by optimizing and adjusting the basic resource scheduling model (the first basic resource scheduling model and the second basic resource scheduling model), the computing power resource scheduling level can be continuously improved, and the efficiency and accuracy of resource scheduling in the computing power network can be further improved, which is not limited to the embodiments of the present invention.

[0140] It can be understood that the target resource scheduling model is determined from all the initial resource scheduling models, and the initial resource scheduling model is obtained by adjusting the basic resource scheduling model. Therefore, the model structures of the target resource scheduling model, the initial resource scheduling model, and the basic resource scheduling model are the same, that is, the model structures of the initial resource scheduling model and the basic resource scheduling model are the model structures of the target resource scheduling model introduced above.

[0141] It should be noted that the computing power resource scheduling tasks in the above computing power resource scheduling task information represent the computing power resource scheduling tasks required to execute the workload, which is not limited in the embodiment of the present invention.

[0142] It should be noted that the task types of the computing resource scheduling tasks in the above computing resource scheduling task information may be consistent or inconsistent, and the computing resource scheduling tasks are independent of each other, which is not limited in the embodiment of the present invention.

[0143] It can be seen that implementing the computing power network scheduling method described in the embodiment of the present invention is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0144] Embodiment 2

[0145] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of a computing power network scheduling device based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a management system, such as for managing a local server or a cloud server, etc., which is not limited in the embodiments of the present invention. Figure 3 As shown, the device may include:

[0146] Acquisition module 201, used to acquire computing power network environment information;

[0147] The first determination module 202 is used to determine the target scheduling model information based on the computing power network environment information and the initial scheduling model information; the target scheduling model information includes M target resource scheduling models; the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model;

[0148] The second determination module 202 is used to determine the target computing power network resource scheduling result information based on the target scheduling model information and the computing power network environment information; the target computing power network resource scheduling result information includes the computing power scheduling type and computing power parameter information.

[0149] It can be seen that implementation Figure 3 The computing power network scheduling device described is conducive to improving the efficiency and accuracy of computing power network resource scheduling, thereby improving the user experience.

[0150] In another optional embodiment, Figure 3 As shown, the second determination module 202 determines the target computing power network resource scheduling result information based on the target scheduling model information and the computing power network environment information, including:

[0151] Determine whether there is a first target resource scheduling model among all target resource scheduling models in the target scheduling model information, and obtain an existence determination result;

[0152] When the judgment result is yes, it is judged whether the attribute priority corresponding to the environment attribute information in the computing power network environment information is consistent with the model priority of the first target resource scheduling model to obtain a consistent judgment result;

[0153] When the consistent judgment result is yes, the computing power network environment information is processed based on the first target resource scheduling model to obtain the target computing power network resource scheduling result information;

[0154] When the consistent judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information;

[0155] When the judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information.

[0156] It can be seen that implementation Figure 3 The computing power network scheduling device described is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0157] In yet another optional embodiment, Figure 3 As shown, the second determination module 202 processes the computing power network environment information based on the first target resource scheduling model to obtain target computing power network resource scheduling result information, including:

[0158] The computing power network environment information is converted and processed to obtain the computing power network space state information; the computing power network space state information represents the data information of the multi-dimensional tensor structure;

[0159] The computing power network space status information is processed by the first resource scheduling model in the first target resource scheduling model to obtain the target computing power network resource scheduling result information; the first resource scheduling model includes a first sub-resource scheduling model, a second sub-resource scheduling model and a third sub-resource scheduling model; the second sub-resource scheduling model and the third sub-resource scheduling model are connected in parallel to the first sub-resource scheduling model.

[0160] It can be seen that implementation Figure 3The computing power network scheduling device described is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0161] In yet another optional embodiment, Figure 3 As shown, the second determination module 202 processes the computing power network space state information using the first resource scheduling model in the first target resource scheduling model to obtain target computing power network resource scheduling result information, including:

[0162] The computing power network space state information is convolved N times using the first sub-resource scheduling model to obtain first state feature information; N is a positive integer not less than 2; the N convolutions are implemented based on the convolution layer in the first sub-resource scheduling model;

[0163] Using the first sub-resource scheduling model, a spatial pyramid pooling operation is performed on the first state feature information to obtain second state feature information; the spatial pyramid pooling operation is implemented based on the pooling layer in the first sub-resource scheduling model;

[0164] The second state characteristic information is fully connected by using the first sub-resource scheduling model to obtain third state characteristic information; the full connection processing is implemented based on the full connection layer in the first sub-resource scheduling model;

[0165] The third state feature information is convolved using the second sub-resource scheduling model to obtain computing power parameter information;

[0166] The third state characteristic information is processed using the third sub-resource scheduling model to obtain the computing power scheduling type.

[0167] It can be seen that implementation Figure 3 The computing power network scheduling device described is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0168] In yet another optional embodiment, Figure 3 As shown, the second determination module 202 processes the third state characteristic information using the third sub-resource scheduling model to obtain a computing power scheduling type, including:

[0169] The third state feature information is converted for the first time using the third sub-resource scheduling model to obtain feature vector information; the feature vector information represents data information with a dimension of 1×256; the first conversion processing of the third state feature information is implemented based on a fully connected neural network in the third sub-resource scheduling model;

[0170] The feature vector information is converted for the second time using the third sub-resource scheduling model to obtain computing power resource scheduling distribution information; the resource scheduling distribution value in the computing power resource scheduling distribution information represents the distribution of computing power resources in the computing power network; the feature vector information is converted for the second time based on the distribution conversion model in the third sub-resource scheduling model;

[0171] Based on the resource scheduling relationship table, the computing power resource scheduling probability information is matched and screened to obtain the computing power scheduling type; the resource scheduling relationship table represents the corresponding relationship between the computing power scheduling type corresponding to the resource scheduling distribution value and the resource scheduling distribution threshold.

[0172] It can be seen that implementation Figure 3 The computing power network scheduling device described is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0173] In yet another optional embodiment, Figure 3 As shown, after the second determination module 202 processes the computing power network space state information using the first resource scheduling model in the first target resource scheduling model to obtain the target computing power network resource scheduling result information, the method further includes:

[0174] Obtain computing power resource usage feedback information corresponding to the target computing power network resource scheduling result information;

[0175] Calculate the computing power resource usage feedback information based on the computing power usage evaluation formula to obtain the usage evaluation value;

[0176] The calculation power usage evaluation formula is:

[0177] ;

[0178] In the formula, To use the evaluation value, , and They are the computing power resources in the first computing power parameter information, the computing power resources in the second computing power parameter information, and the computing power resources in the The number of computing resources in the computing power parameter information, , , , and They are the first evaluation coefficient, the second evaluation coefficient, the first weight coefficient, the second weight coefficient and the Weight coefficient, For the Weight coefficient, To use the first computing power parameter information The estimated value of computing resources. To use the second computing power parameter information The estimated value of computing resources. To use The computing power parameter information The estimated value of computing resources. To use the first computing power parameter information The estimated value of computing resources. To use the second computing power parameter information The estimated value of computing resources. To use The computing power parameter information The estimated value of a computing resource;

[0179] Using the second resource scheduling model in the first target resource scheduling model to perform error calculation processing on the usage evaluation value to obtain a usage error evaluation value;

[0180] The first resource scheduling model is updated using the usage error evaluation value.

[0181] Exemplarily, after the first resource scheduling model is updated using the error evaluation value, an overall optimization goal can also be constructed, and the target computing power network resource scheduling result information can be optimized based on the overall optimization goal to obtain the final target computing power network resource scheduling result information, that is, the overall optimization goal is solved using an optimization algorithm or other solving algorithm to obtain the final computing power scheduling type and computing power parameter information. In the solving process, the computing power network space state information is processed using the updated first resource scheduling model to obtain the target computing power network resource scheduling result information, so as to further improve the accuracy and rationality of the computing power scheduling type and computing power parameter information. The target problem of reinforcement learning is formulated as a Markov decision process (MDP), which consists of a 6-tuple Definition, where and Represent the state space, action space, state transition probability matrix and initial state distribution respectively. R represents the reward function, represents the discount factor. Assume that in the network scenario, each device on the network initiates at most one computing task at the same time. The state space, action space and reward function of the MDP are defined as follows: The state space is , action space , Indicates the task starts from point Scheduled to a service node If this action does not exist, The reward function defines the task set as , where N is the total number of tasks. The optimization goals are as follows.

[0182] The overall optimization goal is:

[0183] ;

[0184] ;

[0185] ;

[0186] in, represents the delay weight coefficient, represents the distance weight coefficient, Represents the computing power weight coefficient, Represents a task scheduling action. Indicates the task starts from point Scheduled to a service node If this action does not exist, , Indicates the total number of nodes, including the starting point of the task and the end point of the calculation implementation on the computing power network. Indicates the maximum length of the queue buffer. Representation Task From point is scheduled, and the order at the target node is , Indicates distance, Indicates the task point The communication distance from the task to the scheduling computing center, Indicates the link transmission delay, Indicates the waiting delay. Indicates execution delay.

[0187] It should be noted that when scheduling a task, if the task needs to be scheduled to other nodes, the scheduling delay is the sum of the link transmission delay, waiting delay and execution delay. If the task does not need to be scheduled to other nodes, the scheduling delay is the sum of the waiting delay and execution delay. The waiting delay is the time to wait for the service node to start executing the task, and the execution delay is the time required for the service node to execute the task. The expression is:

[0188] ;

[0189] ;

[0190] ;

[0191] ;

[0192] in, Indicates the link transmission delay, Representation Task The capacity, Indicates that the slave service node To the service node The propagation rate, Indicates the waiting delay. Indicates that the task starts from the task point Scheduled to a service node If this action does not exist, , Indicates execution delay, Representation Task The current service node is sorted as The capacity, Represents a service node The calculation rate, To calculate the transmission power of the task, is the channel gain, is the channel transmission distance, represents the distance attenuation factor, Represents the background noise of the channel.

[0193] It can be seen that implementation Figure 3 The computing power network scheduling device described is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0194] In yet another optional embodiment, Figure 3 As shown, the initial scheduling model information is obtained based on the following method:

[0195] Obtain computing resource scheduling task information;

[0196] Decompose the computing resource scheduling task information to obtain target scheduling task information; the target scheduling task information includes L target scheduling tasks; L is a positive integer not less than 2;

[0197] A scheduling task tree is constructed based on the task relationship of each target scheduling task in the target scheduling task information; the scheduling task tree includes a plurality of task nodes; each task node corresponds to a target scheduling task;

[0198] Based on the basic scheduling model information, each task node in the scheduling task tree is subjected to model adjustment processing to obtain initial scheduling model information; the initial scheduling model information includes several initial scheduling models; each task node corresponds to K initial scheduling models; K is a positive integer not less than 1.

[0199] It can be seen that implementation Figure 3The computing power network scheduling device described is conducive to improving the efficiency and accuracy of resource scheduling in the computing power network, thereby improving the user experience.

[0200] Embodiment 3

[0201] See also Figure 4 , Figure 4 It is a structural diagram of another computing power network scheduling device based on deep reinforcement learning disclosed in an embodiment of the present invention. Figure 4 The described device can be applied to a management system, such as for managing a local server or a cloud server, etc., which is not limited in the embodiments of the present invention. Figure 4 As shown, the device may include:

[0202] A memory 301 storing executable program codes;

[0203] a processor 302 coupled to the memory 301;

[0204] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the computing power network scheduling method described in the first embodiment.

[0205] Embodiment 4

[0206] An embodiment of the present invention discloses a computer-readable storage medium that stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the computing power network scheduling method described in Embodiment 1.

[0207] Embodiment 5

[0208] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the computing power network scheduling method described in Example 1.

[0209] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0210] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0211] Finally, it should be noted that the computing power network scheduling method, device and medium based on deep reinforcement learning disclosed in the embodiments of the present invention are only the preferred embodiments of the present invention, which are only used to illustrate the technical scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that it is still possible to modify the technical schemes recorded in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical schemes deviate from the spirit and scope of the technical schemes of the various embodiments of the present invention.

Claims

1. A computing power network scheduling method based on deep reinforcement learning, characterized in that: The computing power network scheduling method includes: Obtain computing power network environment information; Based on the computing power network environment information and the initial scheduling model information, target scheduling model information is determined; the target scheduling model information includes M target resource scheduling models, and the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model; Based on the target scheduling model information and the computing power network environment information, determine the target computing power network resource scheduling result information; the target computing power network resource scheduling result information includes computing power scheduling type and computing power parameter information; The determining target computing power network resource scheduling result information based on the target scheduling model information and the computing power network environment information includes: Determine whether there is a first target resource scheduling model among all the target resource scheduling models in the target scheduling model information, and obtain an existence determination result; When the existence judgment result is yes, determine whether the attribute priority corresponding to the environment attribute information in the computing power network environment information is consistent with the model priority of the first target resource scheduling model to obtain a consistent judgment result; When the consistency judgment result is yes, the computing power network environment information is processed based on the first target resource scheduling model to obtain target computing power network resource scheduling result information; When the consistency judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information; When the existence judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information.

2. The computing power network scheduling method according to claim 1, characterized in that: The processing of the computing power network environment information based on the first target resource scheduling model to obtain target computing power network resource scheduling result information includes: The computing power network environment information is converted and processed to obtain computing power network space state information; the computing power network space state information represents data information of a multi-dimensional tensor structure; The computing power network space status information is processed by the first resource scheduling model in the first target resource scheduling model to obtain the target computing power network resource scheduling result information; the first resource scheduling model includes a first sub-resource scheduling model, a second sub-resource scheduling model and a third sub-resource scheduling model; the second sub-resource scheduling model and the third sub-resource scheduling model are connected in parallel to the first sub-resource scheduling model.

3. The computing power network scheduling method according to claim 2, characterized in that: The using the first resource scheduling model in the first target resource scheduling model to process the computing power network space state information to obtain target computing power network resource scheduling result information includes: Using the first sub-resource scheduling model, the computing power network space state information is subjected to N convolution processing to obtain first state feature information; N is a positive integer not less than 2; the N convolutions are implemented based on the convolution layer in the first sub-resource scheduling model; Using the first sub-resource scheduling model to perform a spatial pyramid pooling operation on the first state feature information to obtain second state feature information; the spatial pyramid pooling operation is implemented based on a pooling layer in the first sub-resource scheduling model; Using the first sub-resource scheduling model to perform full connection processing on the second state characteristic information to obtain third state characteristic information; the full connection processing is implemented based on the full connection layer in the first sub-resource scheduling model; Using the second sub-resource scheduling model to perform convolution processing on the third state feature information to obtain the computing power parameter information; The third state characteristic information is processed using the third sub-resource scheduling model to obtain the computing power scheduling type.

4. The computing power network scheduling method according to claim 3 is characterized in that: The using the third sub-resource scheduling model to process the third state characteristic information to obtain the computing power scheduling type includes: The third state feature information is converted for the first time using the third sub-resource scheduling model to obtain feature vector information; the feature vector information represents data information with a dimension of 1×256; the first conversion processing of the third state feature information is implemented based on a fully connected neural network in the third sub-resource scheduling model; The feature vector information is subjected to a second conversion process using the third sub-resource scheduling model to obtain computing power resource scheduling distribution information; the resource scheduling distribution value in the computing power resource scheduling distribution information represents the distribution of computing power resources in the computing power network; the second conversion process of the feature vector information is performed based on the distribution conversion model in the third sub-resource scheduling model; The computing power resource scheduling probability information is matched and screened based on the resource scheduling relationship table to obtain the computing power scheduling type; the resource scheduling relationship table represents the corresponding relationship between the computing power scheduling type corresponding to the resource scheduling distribution value and the resource scheduling distribution threshold.

5. The computing power network scheduling method according to claim 2, characterized in that: After the computing power network space state information is processed by using the first resource scheduling model in the first target resource scheduling model to obtain target computing power network resource scheduling result information, the method further includes: Obtain computing power resource usage feedback information corresponding to the target computing power network resource scheduling result information; Calculating the computing power resource usage feedback information based on a computing power usage evaluation formula to obtain a usage evaluation value; The computing power usage evaluation formula is: ； In the formula, To use the evaluation value, , and They are the computing power resources in the first computing power parameter information, the computing power resources in the second computing power parameter information, and the computing power resources in the The number of computing resources in the computing power parameter information, , , , and They are the first evaluation coefficient, the second evaluation coefficient, the first weight coefficient, the second weight coefficient and the Weight coefficient, For the Weight coefficient, To use the first computing power parameter information The estimated value of computing resources, To use the second computing power parameter information The estimated value of computing resources, To use The computing power parameter information The estimated value of computing resources, To use the first computing power parameter information The estimated value of computing resources, To use the second computing power parameter information The estimated value of computing resources, To use The computing power parameter information The estimated value of a computing resource; Using the second resource scheduling model in the first target resource scheduling model to perform error calculation processing on the usage evaluation value to obtain a usage error evaluation value; The first resource scheduling model is updated using the usage error evaluation value.

6. The computing power network scheduling method according to claim 1, characterized in that: The initial scheduling model information is obtained based on the following method: Obtain computing resource scheduling task information; Decomposing and processing the computing power resource scheduling task information to obtain target scheduling task information; The target scheduling task information includes L target scheduling tasks; L is a positive integer not less than 2; Constructing a scheduling task tree based on the task relationship of each of the target scheduling tasks in the target scheduling task information; The scheduling task tree includes a plurality of task nodes; each of the task nodes corresponds to one of the target scheduling tasks; Based on the basic scheduling model information, model adjustment processing is performed on each task node in the scheduling task tree to obtain the initial scheduling model information; the initial scheduling model information includes several initial scheduling models; each task node corresponds to K initial scheduling models; K is a positive integer not less than 1.

7. The computing power network scheduling method according to claim 5, characterized in that: After the first resource scheduling model is updated by using the usage error evaluation value, the method further includes: Constructing an overall optimization goal, and optimizing the target computing power network resource scheduling result information based on the overall optimization goal to obtain the final target computing power network resource scheduling result information; The overall optimization goal is: ； in, represents the delay weight coefficient, represents the distance weight coefficient, Represents the computing power weight coefficient, Represents a task scheduling action. Indicates the task starts from point Scheduled to a service node If this action does not exist, , Indicates the total number of nodes, including the starting point of the task and the end point of the calculation implementation on the computing power network. Indicates the maximum length of the queue buffer. Indicates the task From point is scheduled, and the order at the target node is , Indicates distance.

8. A computing power network scheduling device based on deep reinforcement learning, characterized in that: The device comprises: The acquisition module is used to obtain computing power network environment information; A first determination module is used to determine target scheduling model information based on the computing power network environment information and the initial scheduling model information; the target scheduling model information includes M target resource scheduling models, and the target resource scheduling model includes a first target resource scheduling model and / or a second target resource scheduling model; A second determination module is used to determine target computing power network resource scheduling result information based on the target scheduling model information and the computing power network environment information; the target computing power network resource scheduling result information includes computing power scheduling type and computing power parameter information; The second determining module is specifically used to implement: Determine whether there is a first target resource scheduling model among all the target resource scheduling models in the target scheduling model information, and obtain an existence determination result; When the existence judgment result is yes, determine whether the attribute priority corresponding to the environment attribute information in the computing power network environment information is consistent with the model priority of the first target resource scheduling model to obtain a consistent judgment result; When the consistency judgment result is yes, the computing power network environment information is processed based on the first target resource scheduling model to obtain target computing power network resource scheduling result information; When the consistency judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information; When the existence judgment result is no, the computing power network environment information is processed based on the second target resource scheduling model to obtain the target computing power network resource scheduling result information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to use the computing power network scheduling method as described in any one of claims 1-7.

Images