Computing power deconstruction strategy generation method and device, equipment and medium

Abstract

The application provides a computing power disintegration strategy generation method and device, equipment and a medium, comprising: obtaining a scheduling queue comprising a plurality of sub-tasks of a componentized task; wherein the sub-tasks are obtained by cutting the componentized task according to computing power requirements; inputting the scheduling queue into a deep reinforcement learning model to obtain a target computing power disintegration strategy of the componentized task; wherein the deep reinforcement learning model is used to output a target computing power disintegration strategy with the shortest task execution time from a plurality of computing power disintegration strategies according to a network environment corresponding to the componentized task, wherein the computing power disintegration strategy refers to a strategy for allocating a network node executing each sub-task of the componentized task. The embodiment of the application reduces the granularity of the task object by dividing the componentized task into a plurality of sub-tasks, effectively saves the total execution time and reduces the occupied resources by comprehensively considering the convergence performance, task execution time and resource occupancy rate through the deep reinforcement learning algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing, and in particular to a computing power disintegration strategy generation method, device, equipment and medium. BACKGROUND

[0002] In the existing computing power network environment, there are three types of nodes: cloud data center, edge cloud and intelligent terminal. The computing resources of heterogeneous nodes are different, and the links between nodes have certain transmission bandwidth and transmission delay. In the real environment, the computing offloading is affected by many factors such as external environment, hardware and software environment, or user personality, such as internal and external interference of wireless channel, performance of terminal edge and cloud, and user personalized use. These prerequisites make it difficult and important to develop a reasonable and adaptive computing power disintegration strategy to the dynamic changes of the environment. Because the computing power provided by terminal devices, edge nodes, core nodes and other levels of computing power nodes has great differences, and with the operation of business and network, the computing power resources will more and more show the fragmentation trend. However, in the existing technology, it is difficult to match the computing task with the nodes that adapt to the computing power demand in the computing power disintegration process, which is easy to cause the task execution time to be too long. SUMMARY

[0003] The purpose of the present application is to provide a computing power disintegration strategy generation method, device, equipment and medium, which is used to solve the problem that it is difficult to match the computing task with the nodes that adapt to the computing power demand in the computing power disintegration process in the prior art, which is easy to cause the task execution time to be too long.

[0004] To achieve the above-mentioned purpose, the embodiments of the present application provide a computing power disintegration strategy generation method, which comprises:

[0005] obtaining a scheduling queue comprising a plurality of subtasks of a componentized task; wherein the subtasks are obtained by cutting the componentized task according to computing power demand;

[0006] inputting the scheduling queue into a deep reinforcement learning model to obtain a target computing power disintegration strategy of the componentized task; wherein the deep reinforcement learning model is used to output the target computing power disintegration strategy with the shortest task execution time among a plurality of computing power disintegration strategies according to the network environment corresponding to the componentized task, wherein the computing power disintegration strategy refers to a strategy for allocating a network node for executing each subtask of the componentized task.

[0007] Optionally, the computing power disintegration strategy generation method, wherein the obtaining a scheduling queue comprising a plurality of subtasks of a componentized task comprises:

[0008] obtaining a plurality of levels; wherein the levels are obtained by topologically sorting and layering a directed acyclic graph (DAG) corresponding to the componentized task;

[0009] ranking the levels; wherein the fewer the number of sub-tasks of the componentized task contained in the level, the higher the priority of the level;

[0010] ranking the sub-tasks in the level; wherein the greater the resource requirement of the sub-task, the higher the priority;

[0011] obtaining a scheduling queue according to the sub-tasks arranged in order of priority from high to low.

[0012] Optionally, the method, wherein the deep reinforcement learning model comprises an environment module and a deep Q network module;

[0013] Before the inputting the scheduling queue into the deep reinforcement learning model and obtaining the computing power disassembly strategy with the shortest execution time of the componentized task, the method further comprises:

[0014] initializing a simulation environment, hyperparameters and a Thompson sampling action exploration strategy;

[0015] establishing the environment module of the componentized task according to the simulation environment, the hyperparameters and the Thompson sampling action exploration strategy; wherein the environment module obtains a reward value and a network environment state according to a current network environment and an input action, and sends the reward value and the network environment state to the deep Q network module, the reward value is inversely proportional to the execution time of the sub-task, and the action is an action of allocating the sub-task to the network nodes in the network;

[0016] establishing the deep Q network module; wherein the deep Q network module is used to obtain the action with the maximum value according to the network environment state, the value is obtained by fitting the network environment state through a fully connected layer by the deep Q network module;

[0017] training the deep Q network module according to the environment module to obtain the deep reinforcement learning model.

[0018] Optionally, the method, wherein the initializing a simulation environment, hyperparameters and a Thompson sampling action exploration strategy comprises:

[0019] initializing the simulation environment according to data information of the componentized task; wherein the data information comprises at least one of the number of sub-tasks, the number of instructions of the sub-tasks, data dependencies among a plurality of sub-tasks, the number of network nodes, the number of instructions executed per second by the network nodes, transmission bandwidth among a plurality of network nodes, and transmission delay among the network nodes;

[0020] initializing the hyperparameters; wherein the hyperparameters include at least one of a reward control factor and a punishment control factor of a reward function, and an epsilon value of an epsilon greedy algorithm;

[0021] initializing the Thompson sampling action exploration strategy, and obtaining the action with the largest random number; wherein the random number is obtained according to a probability distribution corresponding to parameters a and β of the action, the parameter a represents a number of times that a task execution time obtained after the subtask is allocated to the network node is less than or equal to a preset time length, and the parameter β represents a number of times that the task execution time obtained after the subtask is allocated to the network node is greater than the preset time length.

[0022] Optionally, the method, wherein, after the action with the largest random number is obtained, the method further includes:

[0023] determining whether the reward value of the action is a maximum value, and obtaining a first determination result;

[0024] in a case where the first determination result is yes, increasing a value of the parameter a by 1;

[0025] in a case where the first determination result is no, increasing a value of the parameter β by 1.

[0026] Optionally, the method, wherein the environment module of the componentized task is established according to the simulation environment, the hyperparameters, and the Thompson sampling action exploration strategy, and includes:

[0027] in a case where the subtask needs to be scheduled to multiple network nodes, obtaining environment state information according to the simulation environment, and generating a state submodule; wherein the environment state information includes at least one of a serial number d of a current subtask to be scheduled, node information R of the network node, demand information V of the subtask, and corresponding relationship information M between the subtask that has been allocated and the network node; t t

[0028] obtaining a reward submodule according to the hyperparameters and the Thompson sampling action exploration strategy reward and punishment function; wherein a value of the punishment function is -β;

[0029] obtaining an action submodule according to the network node; wherein a number of the network nodes is the same as a number of the actions;

[0030] establishing the environment module according to the state submodule, the reward submodule, and the action submodule.

[0031] Optionally, the method, wherein an action space of the action submodule is a set of numbers of the network nodes.​​

[0032] Optionally, the method, wherein, after the deep Q network module is established, the method further comprises:

[0033] According to the deep Q network module, the value of a plurality of actions under each network environment state is obtained.

[0034] The action with the highest value under the current network environment state is determined, and the corresponding reward value is output to the environment module.

[0035] Optionally, the method, wherein, the deep Q network module is trained according to the environment module to obtain the deep reinforcement learning model, comprising:

[0036] According to the environment module, the current network environment state is obtained.

[0037] The action is selected by an exploration strategy of an epsilon value decay greedy algorithm, and an action value corresponding to the action is obtained.

[0038] According to the action value, the resources of the network node are allocated to the corresponding subtask, and the environment feedback and the next network environment state are obtained.

[0039] Randomly extract a preset data amount of data from a memory unit to calculate a loss, wherein the memory unit is used to store the environment feedback and the network environment state, and the preset data amount is a minibatch.

[0040] Repeat the above steps until the number of repetitions reaches a preset value, and obtain the deep reinforcement learning model.

[0041] In order to achieve the above purpose, the embodiment of the application further provides a computing power deconstruction strategy generation device, comprising:

[0042] A first obtaining module is configured to obtain a scheduling queue of a plurality of subtasks of a componentized task, wherein the subtasks are obtained by cutting the componentized task according to computing power requirements.

[0043] A second obtaining module is configured to input the scheduling queue into a deep reinforcement learning model to obtain a target computing power deconstruction strategy of the componentized task, wherein the deep reinforcement learning model is configured to output the target computing power deconstruction strategy with the shortest task execution time among a plurality of computing power deconstruction strategies according to a network environment corresponding to the componentized task, and the computing power deconstruction strategy refers to a strategy of allocating a network node for executing each subtask of the componentized task.

[0044] To achieve the above object, the embodiment of the present application further provides a network device, comprising a transceiver, a processor, a memory and a program or instructions stored in the memory and executable on the processor, wherein the processor implements the computing power deconstruction strategy generation method as described above when executing the program or instructions.

[0045] To achieve the above object, the embodiment of the present application further provides a readable storage medium having a program or instructions stored thereon, wherein the program or instructions implement the steps in the computing power deconstruction strategy generation method as described above when executed by a processor.

[0046] To achieve the above object, the embodiment of the present application further provides a computer program product comprising computer instructions, wherein the computer instructions implement the steps in the computing power deconstruction strategy generation method as described above when executed by a processor.

[0047] The beneficial effects of the above technical solutions of the present application are as follows:

[0048] The embodiment of the present application inputs the scheduling queue of multiple sub-tasks including the componentized task into a deep reinforcement learning model, obtains a target computing power deconstruction strategy with the shortest task execution time among multiple computing power deconstruction strategies, thereby avoiding that the computing power node cannot carry larger business demands, reducing the granularity of the task object by dividing the componentized task into multiple sub-tasks, and effectively saving the total execution time and reducing the occupied resources by comprehensively considering the convergence performance, task execution time and resource occupancy rate through the deep reinforcement learning algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The figure is a schematic diagram of the computing power deconstruction strategy generation method according to the embodiment of the present application.

[0050] Figure 2 The figure is a hierarchical diagram of the componentized task of the computing power deconstruction strategy generation method according to the embodiment of the present application.

[0051] Figure 3 The figure is a structural schematic diagram of the deep reinforcement learning model of the computing power deconstruction strategy generation method according to the embodiment of the present application.

[0052] Figure 4 The figure is a schematic diagram of the computing power deconstruction strategy generation device according to the embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0054] It should be understood that the reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0055] In various embodiments of the present application, it should be understood that the size of the serial number of the following processes does not mean the order of execution, and the execution order of the processes should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0056] In addition, the terms "system" and "network" are often used interchangeably herein.

[0057] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that the determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0058] For the convenience of understanding, some of the contents related to the embodiments of the present application are described as follows:

[0059] As shown in the figure, a computing power deconstruction strategy generation method of an embodiment of the present application, comprising: Figure 1

[0060] S10, obtaining a scheduling queue comprising a plurality of subtasks of a componentized task; wherein the subtasks are obtained by cutting the componentized task according to computing power requirements;

[0061] It should be noted that there is a dependency relationship between each of the subtasks of the componentized task, and a subtask can only start execution after the corresponding parent task is completed and the result is received. The componentized task can be modeled by a directed acyclic graph (DAG), and the node value represents the computing requirement of the subtask. The componentized task is preprocessed by using a priority allocation algorithm based on topological sorting, and the subtasks are sorted according to the priority of the subtasks first, because the greater the priority of the subtask, the more obvious the optimization of the network node pair delay when the subtask is allocated to the resource, which can greatly improve the exploration efficiency of deep reinforcement learning and speed up the convergence of the algorithm.

[0062] ​S20, inputting the scheduling queue into a deep reinforcement learning model to obtain a target computing power disassembly strategy of the componentized task; wherein the deep reinforcement learning model is used to output the target computing power disassembly strategy with the shortest task execution time from a plurality of computing power disassembly strategies according to a network environment corresponding to the componentized task, wherein the computing power disassembly strategy refers to a strategy of assigning each subtask of the componentized task to a network node for executing the subtask.

[0063] In this embodiment, by inputting the scheduling queue including a plurality of subtasks of the componentized task into the deep reinforcement learning model, the target computing power disassembly strategy with the shortest task execution time from a plurality of computing power disassembly strategies is obtained, thereby avoiding that a computing power node cannot bear a larger business demand, reducing the granularity of a task object by dividing the componentized task into a plurality of subtasks, and effectively saving the total execution time and reducing the occupied resources by comprehensively considering the convergence performance, task execution time and resource occupancy rate through the deep reinforcement learning algorithm.

[0064] Optionally, the computing power disassembly strategy generation method, wherein the step S10 comprises:

[0065] obtaining a plurality of levels; wherein the levels are obtained by topologically sorting and layering a directed acyclic graph (DAG) corresponding to the componentized task;

[0066] It should be noted that, as shown in Figure 2 provides an example, six subtasks are divided into four layers, and the subtasks in each layer are independent of each other and have no dependency relationship.

[0067] sorting the levels; wherein the fewer the number of subtasks of the componentized task in the level, the higher the priority of the level;

[0068] It should be noted that when a layer has only one subtask, the quality of the resources of the network node allocated to the subtask has a direct impact on the execution time of the layer. Therefore, the fewer the number of subtasks in the level, the higher the priority;

[0069] sorting the subtasks in the level; wherein the greater the resource demand of the subtask, the higher the priority;

[0070] It should be noted that when the subtasks in the same level are executed in parallel, the longer the execution time of the subtask, the greater the impact of the subtask on the total execution time of the layer. Therefore, when there are a plurality of subtasks in the same level, the subtask with the greater resource demand is considered to have a higher priority.

[0071] obtaining a scheduling queue according to the subtasks arranged in order from high to low priority.

[0072] It should be noted that the sub-tasks are added to the scheduling queue in order of priority from high to low, and are waiting for scheduling.

[0073] In this embodiment, in order to convert the directed acyclic graph representing the componentized task into an input structure of a fully connected neural network, a priority assignment algorithm based on topological sorting is first used to preprocess the componentized task, and the sub-tasks are first put into a scheduling queue in a certain order, which can greatly improve the exploration efficiency of deep reinforcement learning and speed up the convergence of the algorithm.

[0074] Optionally, the method, wherein the deep reinforcement learning model comprises an environment module and a deep Q network module;

[0075] Before the step S20, the method further comprises:

[0076] Initializing the simulation environment, hyperparameters and Thompson sampling action exploration strategy;

[0077] According to the simulation environment, the hyperparameters and the Thompson sampling action exploration strategy, the environment module of the componentized task is established; wherein the environment module obtains a reward value and a network environment state according to a current network environment and an input action, and sends the reward value and the network environment state to the deep Q network module, the reward value is inversely proportional to the execution time of the sub-task, and the action is the behavior of assigning the sub-task to the network node in the network;

[0078] The deep Q network module is established; wherein the deep Q network module is used to obtain the action with the maximum value according to the network environment state, and the value is obtained by fitting the network environment state through a fully connected layer by the deep Q network module;

[0079] According to the environment module, the deep Q network module is trained to obtain the deep reinforcement learning model.

[0080] In this embodiment, as shown in Figure 3 The deep reinforcement learning model comprises the environment module and the deep Q network module for running the componentized task, the environment module is used to calculate the reward value according to the current network environment state and the input action, and transform the network environment state to the next state, and the reward value and the next network environment state are transmitted back to the deep Q network module for training, and the deep Q network module is used to select the action with the highest value from the optional action space according to the current network environment state.

[0081] Optionally, the method, wherein the initializing the simulation environment, the hyperparameters and the Thompson sampling action exploration strategy comprises:

[0082] initializing the simulation environment according to the data information of the componentized task; wherein the data information comprises at least one of the number of the subtasks, the number of instructions of the subtasks, the data dependency among the plurality of subtasks, the number of the network nodes, the number of instructions executed per second of the network nodes, the transmission bandwidth among the plurality of network nodes, and the transmission delay among the plurality of network nodes;

[0083] initializing the hyperparameters; wherein the hyperparameters comprise at least one of a reward control factor and a penalty control factor of a reward function, and an ε value of an ε-greedy algorithm;

[0084] initializing the Thompson sampling action exploration strategy to obtain the action with the largest random number; wherein the random number is obtained according to a probability distribution corresponding to a parameter α and a parameter β of the action, the parameter α represents a number of times that a task execution time obtained after the subtasks are allocated to the network nodes is less than or equal to a preset time length, and the parameter β represents a number of times that the task execution time obtained after the subtasks are allocated to the network nodes is greater than the preset time length.

[0085] Optionally, the method, wherein after the obtaining the action with the largest random number, the method further comprises:

[0086] determining whether the reward value of the action is a maximum value to obtain a first determination result;

[0087] in a case where the first determination result is yes, increasing a value of the parameter α by 1;

[0088] in a case where the first determination result is no, increasing a value of the parameter β by 1.

[0089] In this embodiment, the process of Thompson sampling is as follows:

[0090] (1) taking out the parameter α and the parameter β corresponding to each candidate action;

[0091] (2) generating the random number corresponding to the action according to the probability distribution corresponding to the parameter α and the parameter β of the action;

[0092] (3) selecting the action with the largest random number as the output;

[0093] (4) observing the reward value, if the reward value is greater than the maximum value of all the reward values that have been obtained, it is considered that selecting the network node can make the task scheduling result better, the value of the parameter a corresponding to the action is increased by 1, otherwise the value of the parameter β is increased by 1.

[0094] After increasing the Thompson action exploration strategy, better network nodes can be screened out, and the deep reinforcement learning model can be more targeted to explore.

[0095] Optionally, the method, wherein the environment module of the component task is established according to the simulation environment, the hyperparameters and the Thompson sampling action exploration strategy, comprising:

[0096] In the case of needing to schedule the subtasks to multiple network nodes, an environment state information is obtained according to the simulation environment, and a state submodule is generated; wherein the environment state information includes at least one of the following: the serial number d of the current subtask to be scheduled, the node information R of the network node, the demand information V of the subtask and the corresponding relationship information M between the allocated subtask and the network node. t t ;

[0097] It should be noted that in the process of the deep reinforcement learning model perceiving the environment, the environment state information includes the information of the subtask to be allocated and the available resource information in the network, so when the subtask needs to be scheduled to the network node, the environment state information should include the serial number d of the subtask to be scheduled in the scheduling queue, the available network node information R in the network, the demand information V of the subtask and the corresponding relationship information M between the allocated subtask and the network node. t t Specifically represented as: [d, R t , V, M t ].

[0098] A reward submodule is obtained according to the hyperparameters and the Thompson sampling action exploration strategy reward function; wherein the value of the penalty function is -β.

[0099] ​​It should be noted that the design of the reward and punishment function has a great influence on the performance of the deep reinforcement learning algorithm. Because the optimization goal is to minimize, the reward value should be inversely proportional to the total execution time (app_tf) of the task, that is, a / app_tf. At the same time, in order to ensure the dimension of the neural network, the action in each state is the same, but in fact, in a certain state, some network nodes have been allocated to higher priority subtasks, at this time, if the same network node is selected, it will not meet the requirements, so the penalty function is designed, the value of the penalty function is -β. The parameters a and β are two hyperparameters, which are used to adjust the reward and punishment values to a suitable range in order to better train the neural network.

[0100] According to the network node acquisition action submodule; wherein the number of network nodes is the same as the number of actions;

[0101] It should be noted that the action refers to the behavior of allocating the subtask to the available network node, therefore, there should be several actions for several network nodes, and the action space is the union of the numbers of these network nodes.

[0102] According to the state submodule, the reward submodule and the action submodule, the environment module is established;

[0103] It should be noted that, as shown in Figure 3 The environment module includes the state submodule, the reward submodule and the action submodule.

[0104] Optionally, the method, wherein the action space of the action submodule is a set of numbers of the network nodes;

[0105] It should be noted that the action refers to the behavior of allocating the subtask to the available network node, therefore, there should be several actions for several network nodes, and the action space is the union of the numbers of these network nodes: a∈{0,…,K-1}.

[0106] Optionally, the method, wherein after the deep Q network module is established, the method further comprises:

[0107] According to the deep Q network module, the value of a plurality of actions in each network environment state is acquired;

[0108] The action with the highest value in the current network environment state is determined and output to the environment module to obtain the corresponding reward value.

[0109] In the embodiment, the deep Q network module fits the value of different actions under each network environment state. Network parameters are initialized by a random initialization method, the value of the action is set as a random value in advance, and the network environment state is fitted to the value through multiple fully connected layers. The deep Q network module selects the action with the highest value among different values under the current network environment state, and outputs the selected action to the environment module to obtain the corresponding reward value.

[0110] Optionally, the method, wherein the training of the deep Q network module according to the environment module to obtain the deep reinforcement learning model comprises:

[0111] obtaining the current network environment state according to the environment module;

[0112] It should be noted that the deep reinforcement learning model first obtains the network environment state S t In the process of perceiving the environment by the deep reinforcement learning model, the network environment information includes the subtask information to be allocated, the network nodes available in the network, the demand information of the subtasks, and the corresponding relationship information of the allocated subtasks and network nodes: [d, R t , V, M t ].

[0113] The action is selected by an exploration strategy of a greedy algorithm with ε value decay, and the action value corresponding to the action is obtained;

[0114] It should be noted that the action is selected by an exploration strategy of a greedy algorithm with ε value decay, and the deep reinforcement learning model generates a random action value according to an ε probability, and generates the action value according to the deep Q network module according to a 1-ε probability: a t = argmax a Q(φ(S t ), a; θ), and the Thompson sampling action exploration strategy is added thereafter.

[0115] According to the action value, the resources of the network nodes are allocated to the corresponding subtasks, and the environment feedback and the next network environment state are obtained;

[0116] It should be noted that the deep reinforcement learning model allocates the resources of the network nodes to the corresponding subtasks according to the action value, obtains the feedback r t of the environment and the next network environment state S t+1 , and transmits [S t , a t , r t , S t+1store to the memory unit.

[0117] randomly draw a preset data amount of data from the memory unit to calculate the loss; wherein, the memory unit is used to store the environment feedback and the network environment state; the preset data amount is a minibatch;

[0118] It should be noted that, since the training of the neural network requires certain data, the deep reinforcement learning model is allowed to explore the environment for a period of time before learning, and when the current step is greater than the exploration step (step> store), data of a minibatch size is randomly drawn from the memory unit, and the loss is calculated by (y i -Q(φ j , a j ; θ)) 2 calculate the loss, wherein:

[0119]

[0120] γ is a discount factor, which means that the recent income is considered and the long-term income is weakened while ensuring the convergence of the Q network. Then, the deep Q network module is updated by using the gradient descent method.

[0121] repeat the above steps until the number of repetitions reaches a preset value, and obtain the deep reinforcement learning model;

[0122] It should be noted that the predetermined value can be set according to actual conditions and requirements.

[0123] In this embodiment, the deep reinforcement learning model is used to obtain the cloud edge end computing power disintegration strategy method, the environment information is self-learned through reinforcement learning, without relying on artificial experience and rules; the state space is abstracted and represented through the deep neural network; the reasoning stage only depends on the deep neural network, and the reasoning speed is fast. By using the deep reinforcement learning algorithm, the convergence performance, task execution time and resource occupancy are comprehensively considered, the total execution time is effectively saved, and the occupied resources are reduced. By using the preprocessing and Thompson action exploration strategy, the stability and convergence speed of the deep reinforcement learning algorithm are improved.

[0124] As shown in Figure 4 in order to achieve the above purpose, the embodiment of the present application also provides a computing power disintegration strategy generation device, which comprises:

[0125] The first acquisition module 401 is configured to acquire a scheduling queue comprising a plurality of subtasks of a componentized task; wherein, the subtasks are obtained by cutting the componentized task according to computing power requirements;

[0126] The second acquisition module 402 is configured to input the scheduling queue into a deep reinforcement learning model to acquire a target algorithm power disintegration strategy of the componentized task; the deep reinforcement learning model is configured to output, according to a network environment corresponding to the componentized task, the target algorithm power disintegration strategy with the shortest task execution time from a plurality of algorithm power disintegration strategies, and the algorithm power disintegration strategy refers to a strategy of allocating a network node for executing each subtask of the componentized task.

[0127] Optionally, the algorithm power disintegration strategy generation apparatus, wherein the first acquisition module 401 comprises:

[0128] The first acquisition unit is configured to acquire a plurality of levels, wherein the levels are obtained by topologically sorting and layering a directed acyclic graph (DAG) corresponding to the componentized task;

[0129] The first processing unit is configured to sort the levels, wherein the fewer the number of subtasks of the componentized task contained in the level, the higher the priority of the level;

[0130] The second processing unit is configured to sort the subtasks in the levels, wherein the greater the resource demand of the subtask, the higher the priority;

[0131] The second acquisition unit is configured to acquire a scheduling queue according to the subtasks arranged in descending order of priority.

[0132] Optionally, the apparatus, wherein the deep reinforcement learning model comprises an environment module and a deep Q network module;

[0133] The apparatus further comprises:

[0134] The first processing module is configured to initialize a simulation environment, hyperparameters, and a Thompson sampling action exploration strategy;

[0135] The second processing module is configured to establish the environment module of the componentized task according to the simulation environment, the hyperparameters, and the Thompson sampling action exploration strategy; the environment module is configured to acquire a reward value and a network environment state according to a current network environment and an input action, and send the reward value and the network environment state to the deep Q network module, the reward value is inversely proportional to the execution time of the subtask, and the action refers to an action of allocating the subtask to the network node in the network;

[0136] The third processing module is configured to establish the deep Q network module; the deep Q network module is configured to acquire the action with the maximum value according to the network environment state, and the value is obtained by fitting the network environment state through a fully connected layer.

[0137] a third obtaining module, configured to obtain the deep reinforcement learning model by training the deep Q network module according to the environment module.

[0138] Optionally, the apparatus, wherein the first processing module comprises:

[0139] a third processing unit, configured to initialize the simulation environment according to data information of the componentized task; wherein the data information comprises at least one of a number of the subtasks, a number of instructions of the subtasks, data dependencies among the plurality of subtasks, a number of the network nodes, a number of instructions executed per second by the network nodes, a transmission bandwidth among the plurality of network nodes, and a transmission delay among the network nodes;

[0140] a fourth processing unit, configured to initialize the hyperparameters; wherein the hyperparameters comprise at least one of a reward control factor and a penalty control factor of a reward function, and an ε value of an ε greedy algorithm;

[0141] a third obtaining unit, configured to initialize the Thompson sampling action exploration strategy, and obtain the action with the largest random number; wherein the random number is obtained according to a probability distribution corresponding to a parameter α and a parameter β of the action, the parameter α represents a number of times that a task execution time obtained after the subtask is allocated to the network node is less than or equal to a preset time length, and the parameter β represents a number of times that the task execution time obtained after the subtask is allocated to the network node is greater than the preset time length.

[0142] Optionally, the apparatus further comprises:

[0143] a fourth obtaining module, configured to determine whether the reward value of the action is a maximum value, and obtain a first determination result;

[0144] a fourth processing module, configured to, in a case where the first determination result is yes, increase a value of the parameter α by 1;

[0145] a fifth processing module, configured to, in a case where the first determination result is no, increase a value of the parameter β by 1.

[0146] Optionally, the apparatus, wherein the second processing module comprises:

[0147] a fifth processing unit, configured to, in a case where the subtask needs to be scheduled to a plurality of network nodes, obtain environment state information according to the simulation environment, and generate a state submodule; wherein the environment state information comprises a serial number d of a current to-be-scheduled subtask, and node information R of the network nodes tat least one of the following: the demand information V of the subtask, and the correspondence information M of the subtask and the network node that has been allocated t

[0148] a fourth obtaining unit configured to obtain a reward submodule according to the hyperparameter and the Thompson sampling action exploration strategy reward function; wherein the value of the penalty function is -β;

[0149] a fifth obtaining unit configured to obtain an action submodule according to the network node; wherein the number of the network nodes is the same as the number of the actions;

[0150] a sixth processing unit configured to establish the environment module according to the state submodule, the reward submodule and the action submodule.

[0151] Optionally, the apparatus, wherein the action space of the action submodule is a set of numbers of the network nodes.

[0152] Optionally, the apparatus further comprises:

[0153] a fifth obtaining module configured to obtain the value of a plurality of actions under each network environment state according to the deep Q network module;

[0154] a sixth processing module configured to determine the action with the highest value under the current network environment state and output it to the environment module to obtain the corresponding reward value.

[0155] Optionally, the apparatus, wherein the third obtaining module comprises:

[0156] a sixth obtaining unit configured to obtain the current network environment state according to the environment module;

[0157] a seventh obtaining unit configured to select the action by an exploration strategy of a greedy algorithm with ε value decay and obtain the action value corresponding to the action;

[0158] an eighth obtaining unit configured to allocate the resources of the network nodes to the corresponding subtasks according to the action value, obtain the environment feedback and the next network environment state;

[0159] a seventh processing unit configured to randomly extract a preset data amount of data from a memory unit to calculate a loss; wherein the memory unit is configured to store the environment feedback and the network environment state; and the preset data amount is a minibatch;

[0160] a ninth obtaining unit configured to repeat the above steps until the number of repetitions reaches a preset value, and obtain the deep reinforcement learning model.

[0161] ​It should be noted that the above-mentioned device provided by the embodiments of the present application can realize all the method steps achieved by the method embodiments and achieve the same technical effects. Therefore, the same parts and beneficial effects of the method embodiments will not be described in detail.

[0162] To achieve the above object, the embodiments of the present application further provide a network device, comprising a transceiver, a processor, a memory and a program or instructions stored in the memory and executable on the processor; wherein the processor implements the computing power deconstruction strategy generation method as described above when executing the program or instructions.

[0163] To achieve the above object, the embodiments of the present application further provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions are executable on a processor to implement the steps in the computing power deconstruction strategy generation method as described above and achieve the same technical effects. To avoid repetition, details will not be described here.

[0164] To achieve the above object, the embodiments of the present application further provide a computer program product comprising computer instructions, which are executable on a processor to implement the steps in the computing power deconstruction strategy generation method as described above.

[0165] The above exemplary embodiments are described with reference to the drawings, and many different forms and embodiments are possible without departing from the spirit and teachings of the present application. Accordingly, the present application should not be construed as limited to the exemplary embodiments presented herein. Rather, these exemplary embodiments are provided so that this present application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art. In the drawings, the size and relative sizes of components can be exaggerated for clarity. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, a range includes the range's endpoints and any sub-ranges therebetween.

[0166] The above describes the preferred embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for generating computing power deconstruction strategies, characterized in that, include: Obtain a scheduling queue of multiple subtasks including a componentized task; wherein the subtasks are obtained by cutting the componentized task according to the computing power requirements, and the dependencies between the subtasks are represented by a directed acyclic graph (DAG). The step of obtaining the scheduling queue of multiple subtasks includes: Perform topological sorting and layering on the DAG corresponding to the componentized task to obtain multiple levels; The priority of a level and its subtasks is dynamically determined based on the number of subtasks and their resource requirements. The fewer subtasks a level contains, the higher its priority. Within the same level, the greater the resource requirements of a subtask, the higher its priority. A scheduling queue is obtained based on the subtasks arranged in descending order of priority. The scheduling queue is input into a deep reinforcement learning model to obtain the target computing power deconstruction strategy for the componentized task; wherein, the deep reinforcement learning model is used to output the target computing power deconstruction strategy with the shortest task execution time among multiple computing power deconstruction strategies according to the network environment corresponding to the componentized task, wherein, the computing power deconstruction strategy refers to the strategy of allocating a network node to execute the subtask for each subtask of the componentized task. The deep reinforcement learning model includes an environment module and a deep Q-network module; Before inputting the scheduling queue into the deep reinforcement learning model to obtain the computational deconstruction strategy with the shortest execution time for the componentized task, the method further includes: Initialize the simulation environment, hyperparameters, and explore Thompson sampling action strategies; The environment module of the componentized task is established based on the simulation environment, the hyperparameters, and the Thompson sampling action exploration strategy; wherein, the environment module obtains a reward value and a network environment state based on the current network environment and the input action, and sends the reward value and the network environment state to the deep Q network module, the reward value is inversely proportional to the execution time of the sub-task, and the action is the behavior of assigning the sub-task to the network node in the network; The deep Q-network module is established; wherein the deep Q-network module is used to obtain the action with the highest value according to the network environment state, and the value is obtained by the deep Q-network module fitting the network environment state through a fully connected layer; The deep Q-network module is trained based on the environment module to obtain the deep reinforcement learning model.

2. The method according to claim 1, characterized in that, The initialization simulation environment, hyperparameters, and Thompson sampling action exploration strategy include: The simulation environment is initialized based on the data information of the componentized task; wherein, the data information includes at least one of the following: the number of subtasks, the number of instructions of the subtasks, the data dependencies between multiple subtasks, the number of network nodes, the number of instructions executed per second by the network nodes, the transmission bandwidth between multiple network nodes, and the transmission latency between the network nodes. The hyperparameters are initialized; wherein the hyperparameters include at least one of the reward control factor and penalty control factor of the reward function, and the ε value of the ε-greedy algorithm; The Thompson sampling action exploration strategy is initialized to obtain the action with the largest random number; wherein the random number is obtained according to the probability distribution corresponding to the parameters α and β of the action, the parameter α represents the number of times the task execution time obtained after the subtask is assigned the network node is lower than or equal to a preset duration, and the parameter β represents the number of times the task execution time obtained after the subtask is assigned the network node is higher than the preset duration.

3. The method according to claim 2, characterized in that, After the action of obtaining the largest random number, the method further includes: Determine whether the reward value of the action is the maximum value, and obtain the first determination result; If the first determination result is yes, the value of the parameter α is increased by 1; If the first determination result is negative, the value of parameter β is increased by 1.

4. The method according to claim 1, characterized in that, The environment module for establishing the componentized task based on the simulation environment, the hyperparameters, and the Thompson sampling action exploration strategy includes: When it is necessary to schedule the subtask to multiple network nodes, an environment state information is obtained based on the simulation environment to generate a state submodule; wherein, the environment state information includes: the sequence number d of the subtask to be scheduled, and the node information R of the network node. t The subtask requirement information V and the correspondence information M between the assigned subtasks and the network nodes. t At least one of them; The reward submodule is obtained based on the hyperparameters and the Thompson sampling action exploration strategy reward and punishment function; wherein, the value of the punishment function is -β; The action submodule is obtained based on the network nodes; wherein the number of network nodes is the same as the number of actions; The environment module is established based on the state submodule, the reward submodule, and the action submodule.

5. The method according to claim 4, characterized in that, The action space of the action submodule is a set of the network node numbers.

6. The method according to claim 1, characterized in that, After establishing the deep Q-network module, the method further includes: The value of multiple actions in each network environment state is obtained according to the deep Q network module; The action with the highest value under the current network environment state is determined and output to the environment module to obtain the corresponding reward value.

7. The method according to claim 1, characterized in that, The step of training the deep Q-network module based on the environment module to obtain the deep reinforcement learning model includes: The current network environment status is obtained according to the environment module; The action is selected by exploring the strategy through a greedy algorithm with ε-value decay, and the action value corresponding to the action is obtained. Based on the action value, the network node's resources are allocated to the corresponding subtask, and environmental feedback and the next network environment status are obtained. The loss is calculated by randomly sampling a preset amount of data from the memory unit; wherein, the memory unit is used to store the environmental feedback and the network environment state; the preset amount of data is a minibatch; Repeat the above steps until the number of repetitions reaches a preset value to obtain the deep reinforcement learning model.

8. A computing power deconstruction strategy generation device, characterized in that, include: The first acquisition module is used to acquire a scheduling queue of multiple subtasks including a componentized task; wherein the subtasks are obtained by cutting the componentized task according to the computing power requirements, and the dependencies between the subtasks are represented by a directed acyclic graph (DAG). The step of obtaining the scheduling queue of multiple subtasks includes: Perform topological sorting and layering on the DAG corresponding to the componentized task to obtain multiple levels; The priority of a level and its subtasks is dynamically determined based on the number of subtasks and their resource requirements. The fewer subtasks a level contains, the higher its priority. Within the same level, the greater the resource requirements of a subtask, the higher its priority. Obtain the scheduling queue based on the subtasks arranged in descending order of priority; The second acquisition module is used to input the scheduling queue into a deep reinforcement learning model to acquire the target computing power deconstruction strategy of the componentized task; wherein, the deep reinforcement learning model is used to output the target computing power deconstruction strategy with the shortest task execution time among multiple computing power deconstruction strategies according to the network environment corresponding to the componentized task, wherein the computing power deconstruction strategy refers to the strategy of allocating a network node to execute the subtask for each subtask of the componentized task. The deep reinforcement learning model includes an environment module and a deep Q-network module. The device further includes: The first processing module is used to initialize the simulation environment, hyperparameters, and Thompson sampling action exploration strategy. The second processing module is used to establish the environment module of the componentized task based on the simulation environment, the hyperparameters, and the Thompson sampling action exploration strategy; wherein, the environment module obtains a reward value and a network environment state based on the current network environment and the input action, and sends the reward value and the network environment state to the deep Q network module, the reward value is inversely proportional to the execution time of the sub-task, and the action is the behavior of assigning the sub-task to the network node in the network; The third processing module is used to establish the deep Q-network module; wherein the deep Q-network module is used to obtain the action with the highest value according to the network environment state, and the value is obtained by the deep Q-network module fitting the network environment state through a fully connected layer; The third acquisition module is used to train the deep Q-network module according to the environment module to acquire the deep reinforcement learning model.

9. A network device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the computing power deconstruction strategy generation method as described in any one of claims 1-7.

10. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the computing power deconstruction strategy generation method as described in any one of claims 1-7.

11. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps in the computational power deconstruction strategy generation method as described in any one of claims 1-7.

Images