Resource allocation method oriented to dependent tasks

Abstract

The invention relates to a dependent task-oriented resource allocation method, and belongs to the technical field of wireless communication. The method comprises the following steps: S1, in an industrial internet scene, constructing an industrial internet system model comprising a cloud server, an edge server and industrial internet equipment, and modeling a dependency relationship among tasks into a directed acyclic graph; s2, constructing a system total time delay energy consumption weighted sum mathematical model of the industrial internet network; and S3, modeling the problem model as a Markov decision process, and using a deep reinforcement learning (DRL)-based SAC algorithm framework to obtain a task-dependent approximate optimal unloading strategy in an industrial internet IIOT equipment scene with a plurality of edge servers.

Description

Technical Field

[0001] The invention belongs to the technical field of wireless communications and relates to a resource allocation method oriented to dependent tasks. Background Art

[0002] With the rapid development of modern information technology, the manufacturing industry, logistics industry, infrastructure and other industries are undergoing digital transformation, which has led to the rapid development of the Internet of Things (IoT). As an important research field of IoT, Industrial Internet of Things (IIoT) can be used as a key technology to solve the above problems. IIoT is the application of IoT in industrial manufacturing. Sensors, controllers, actuators, IIoT devices, etc. in industrial production environments are all interconnected. Therefore, IIoT has great potential in improving the efficiency, productivity and intelligence of industrial factories.

[0003] IIoT realizes the interconnection of heterogeneous IIoT devices, which can improve manufacturing efficiency and reduce production costs. Since the computing power and battery capacity of the device are limited, IIoT devices are capable of processing some applications with low computing requirements by themselves. However, it is often challenging for IIoT devices to process applications with large computing requirements. In traditional cloud computing solutions, when the processing power of IIoT devices is limited, IIoT devices offload their computing tasks to remote cloud servers for processing, and then return the processed results to the corresponding devices. This relieves the computing power of the devices to a certain extent. With the rapid growth of the number of IIoT devices, the number of corresponding tasks will also increase, so the tasks can be offloaded to cloud servers. However, for delay-sensitive or computationally intensive applications, if the tasks are still offloaded to the cloud, there will be a large transmission delay and may cause network congestion, which will not meet the delay requirements of the tasks and reduce the service quality. This is very dangerous for some special factories that are in operation, and the processing results cannot be returned in time, resulting in unpredictable losses.

[0004] In traditional cloud computing-based solutions, IIoT devices need to offload their tasks to a remote cloud computing server (CCS). Data traffic grows dramatically with the number of IIoT devices and tasks, leading to network congestion, increased task delays, and reduced quality of service (QoS). Fortunately, edge computing deploys computing resources near IIoT devices, providing an effective solution to alleviate these problems. This technology enables IIoT devices to offload their tasks to edge computing servers (ECS) instead of remote CCSs, which can meet the real-time, security, and reliability requirements of IIoT applications that are computationally expensive and latency-sensitive, while also saving energy consumption for devices and effectively alleviating core network congestion. Although edge servers can handle many tasks, they are still lightweight compared to the cloud, so the computing power of a single edge server is also limited. In addition, industrial Internet networks are heterogeneous and the workloads of edge servers vary. Therefore, horizontal collaboration between multiple edge servers is used to balance their workloads to make full use of idle computing resources. Cooperative task offloading and resource allocation are jointly optimized to reduce task delays and improve resource utilization.

[0005] Through the above analysis, it is worth noting that in most application scenarios, computational offloading algorithms are mainly used to process independent tasks without considering when certain tasks depend on certain other tasks. However, in real industrial scenarios, some tasks have complex dependencies and correlations. On the basis of considering collaborative edge computing, by analyzing the characteristics and dependencies of tasks, the computational efficiency of IIoT networks can be better improved. Summary of the invention

[0006] In view of this, an object of the present invention is to provide a resource allocation method oriented to dependent tasks.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A resource allocation method for dependent tasks, the method comprising the following steps:

[0009] S1: Build an industrial Internet system model including cloud servers, edge servers and industrial Internet devices, and describe the dependencies between industrial equipment tasks in a real industrial environment through a directed acyclic graph.

[0010] S2: Construct a weighted and mathematical model of total delay and energy consumption of the industrial Internet system;

[0011] S3: Model the problem model as a Markov decision process and propose a SAC (Soft Actor Critic) algorithm framework based on deep reinforcement learning (DRL).

[0012] Furthermore, in S1, according to the characteristics of cloud servers, edge servers and industrial equipment, an industrial Internet network system architecture is constructed, specifically:

[0013] The system consists of a remote cloud server (CS) c, an SDN controller, M base stations N IIoT devices Composition. Each Bs Each IIoT device has a corresponding edge computing server, which is also denoted as m. If the IIoT device has sufficient computing resources, the entire task will be processed locally. Otherwise, the task will be offloaded to the local Bs for processing, and the local Bs can offload the task to another Bs or cloud center via wired transmission. Each IIoT has a computing-intensive application and decides which devices to perform its tasks. During execution, users can only communicate directly with one Bs. Edge servers collaborate via fiber optic transmission. Due to the competition for computing resources and communication resources between users, the state changes of the system at different times should be considered during the computing offloading process.

[0014] Furthermore, in S2, according to the characteristics of cloud servers, edge servers and industrial Internet devices, an industrial Internet network communication model and a weighted sum of the total system delay energy consumption are constructed, specifically:

[0015] Based on the preparation time of the task and the arrival time of the application, the waiting time v of the task is calculated. n,i is the maximum value of the sum of the completion time and the result output time of all previous computing tasks. Similar to the computing mode on the edge server, the result output time of the computing task can be ignored. Therefore, based on the dependencies between computing tasks, the computing task v n,i The waiting time can be defined as follows:

[0016]

[0017] i'∈pred(i) represents task v n,i The set of prerequisite tasks.

[0018] According to the above analysis of task processing, IIoT device n completes task v n,i The total delay is:

[0019]

[0020] Among them, k = 0 means that the task is processed locally. indicates that the task is processed on the edge server, and k=M+1 indicates that the task is processed on the cloud server. Then, we calculate the task v n,i The actual completion time can be calculated by the following formula:

[0021] FT n,i =RT n,i +t n,i

[0022] Definition 1 {λ} is an indicator function, that is, if the event λ is true, the function value is 1, otherwise, the value is 0. Similarly, the computation task v n,i The energy consumption is:

[0023]

[0024] In this paper, the goal is to find a near-optimal offloading strategy for each application generated by an IIoT device to minimize the average ETC of all users. The ETC of each device is defined as:

[0025]

[0026] Under the energy consumption and delay constraints of IIoT devices, the present invention jointly optimizes the offloading strategy, computing resources and power to minimize the total delay of all device tasks. The scheduling and offloading problem of computing tasks with interdependencies can be formulated as follows:

[0027]

[0028] Furthermore, in S3, the problem model is modeled as a Markov decision process, and a DRL-based SAC algorithm framework is used to obtain the approximate optimal offloading strategy of dependent tasks in a multi-IIoT device scenario with multiple ECSs. At each decision time t, the scheduling agent observes the state of the IIoT environment and the state of the current decision task to select an offloading action and obtain a reward based on a custom goal. The details are as follows:

[0029] Markov decision process state space: At decision time t, the SDN controller collects the state of the environment and represents it as s t ,s t ∈S. It includes edge server status, device status, and task dependencies. The remaining computing resources, bandwidth, and task waiting queue of the edge server, as well as the computing power of the IIoT device and the currently executed tasks are all necessary information for decision making. As for the dependencies between tasks, since it is a data related to the graph, GCN (graph convolutional neural network) is used to extract its task features, and the result is output as a task-level embedding where D is the embedding dimension of each task and I is the number of tasks generated by IIoT device n.

[0030] At each decision moment, the action space includes the current offloading decision, computing resource allocation, and power size.

[0031] In each decision step, there is a task to be decided. Given a state s t , we can obtain the ETC of all devices in the system at time t. After executing the action, the state changes from s t becomes t+1 , the reward at time step t is defined as the negative increment of the sum of ETC of all users in the IIoT system, calculated as:

[0032]

[0033] Since the MDP proposed in this paper shows high-dimensional and continuous state and action spaces, as well as multi-modal rewards (i.e., delay and energy consumption), we use the SAC algorithm to make unloading decisions. SAC adopts MERL to obtain the optimal strategy that maximizes its expected return and expected entropy, which is defined as:

[0034]

[0035] where E[·] represents the expectation of the random variable, H(π(·|s t )) is the state s t The entropy of policy π in . α is the weight of entropy. The soft-Q value function of policy π is given by According to the Bellman equation, it can be expressed as:

[0036]

[0037] The algorithm proposed in the present invention comprises three components: an evaluation network, a target network and an experience revisit pool. The evaluation network comprises a strategy network and a value network, and the target network comprises a target value network and a target strategy network.

[0038] Then, in the soft policy evaluation phase, we define the y The loss function of the parameter y is:

[0039]

[0040] During the training phase, the network parameters are trained by maximizing the policy’s soft state value function

[0041] J(μ)=E[Q y (s t ,a t )-alogp m (a t |s t )]

[0042] Second, the entropy weight a is adjusted using an adaptive gradient-based method. Specifically, a must be large enough to encourage exploration when the optimal behavior is not determined, and gradually decrease as the optimal behavior is determined. Therefore, the weight a can be updated by minimizing the following objective function:

[0043]

[0044] Finally, to enhance learning stability, the parameters of the target network are updated by slowly tracking the parameters of the evaluation network. and Given by:

[0045]

[0046] The beneficial effect of the present invention is that due to the complexity of tasks generated by industrial equipment, the dependency relationship between tasks is focused on. The problem of offloading multiple dependent tasks in end-edge-cloud collaboration is established, the dependency relationship between tasks is modeled as a directed acyclic graph (DAG), and the dependency offloading problem in an IIoT system with multiple IIoT devices and multiple edge servers is modeled as a Markov decision process (MDP). A SAC algorithm based on DRL is proposed. The method provided by the present invention can achieve a more significant improvement in service quality, and a good balance is achieved in terms of cost and risk.

[0047] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0049] Figure 1 It is a network system framework diagram of the present invention;

[0050] Figure 2 This is a flow chart of the task offloading decision and resource allocation according to the present invention. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0053] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0054] See also Figure 1-2 , which is a resource allocation method for dependent tasks, specifically includes the following steps:

[0055] Step 1: Construct an industrial Internet system model including cloud servers, edge servers and industrial Internet devices, and describe the dependencies between industrial equipment tasks in a real industrial environment through a directed acyclic graph. The system considered consists of a remote cloud server (CS) c, an SDN controller, M base stations N IIoT devices Composition. Each Bs Each IIoT device has a corresponding edge computing server, which is also denoted as m. If the IIoT device has sufficient computing resources, the entire task will be processed locally. Otherwise, the task will be offloaded to the local Bs for processing, and the local Bs can offload the task to another Bs or cloud center via wired transmission. Each IIoT has a computing-intensive application and decides which devices to perform its tasks. During execution, users can only communicate directly with one Bs. Edge servers collaborate via fiber optic transmission. Due to the competition for computing resources and communication resources between users, the state changes of the system at different times should be considered during the computing offloading process.

[0056] The applications generated by user n can be modeled as a directed acyclic graph DAGG n =(V n ,E n ), which is divided into I interdependent tasks, where V n = {v n,i |i=1,2,...,I} represents the task of the application. It is the dependency relationship between tasks, indicating that the subsequent task must be started after the predecessor task is completed. Each task is represented by a tuple (d n,i ,c n,i ) represent the data size of the task and the number of CPU cycles required to complete the task. In particular, task V 1,0 and Task V 1,7 They are defined as entry and exit tasks respectively. It should be noted that these two tasks need to be executed locally.

[0057] Step 2: Construct a weighted and mathematical model of total delay and energy consumption of the industrial Internet system

[0058] Step 2.1: IIoT devices and Bs communicate with each other using wireless channels and use To represent the upload task v from IIoT device n to edge server m n,i The uplink rate of . It is assumed that the wireless channels between different edge servers do not affect each other, and Depends on the average channel conditions, the modulation and coding scheme, and the transmit power p n,i According to Shannon's theorem, the wireless communication rate between IIoT device n and edge server m is:

[0059]

[0060] The edge server Bs is connected to the SDN controller through an optical fiber link, and its transmission rate is set to r m,s , SDN control connects to the cloud server through the core backbone network, and its transmission rate is set to r s,c .

[0061] Step 2.2 (local calculation):

[0062] The calculated delay is:

[0063]

[0064] Calculate the energy consumption as:

[0065]

[0066] Step 2.3 (Local Edge Computing):

[0067] Uplink transmission delay of the task:

[0068]

[0069] Computational latency of local edge server processing tasks:

[0070]

[0071] The energy consumption of transmitting the offloaded task from IIoT device n to the local Bsm is:

[0072]

[0073] Step 2.3 (Collaborative Edge Computing):

[0074] From the local BS m to the cooperative BS via a wired link The transmission delay is:

[0075]

[0076] Collaborative ECS The computational delay for processing offloaded tasks is:

[0077]

[0078] The selection of cooperative BS is:

[0079]

[0080] Step 2.4 (Cloud Computing)

[0081] The task transmission delay from the local BS m to CS c is:

[0082]

[0083] The latency of the CS calculation task is:

[0084]

[0085] Step 2.5:

[0086] According to the above analysis of task processing, IIoT device n completes task v n,i The total delay is:

[0087]

[0088] Minimize the average ETC of all users. The ETC of each device is defined as:

[0089]

[0090] Step 2.6:

[0091] The scheduling and offloading problem of computational tasks with interdependencies can be formulated as follows:

[0092]

[0093] Among them, constraint (C1) is the value range of the offloading decision variable. Constraint (C2) indicates that a task can only be calculated by one device. Constraint (C3) indicates the value range of the local ECS and the collaborative ECS to allocate computing resources. Constraint (C4) indicates the power value range of the IIoT device. C5 constrains the energy consumed by the computing task.

[0094] Step 3:

[0095] The problem model of step 2 is modeled as a Markov decision process, and a SAC algorithm framework based on DRL is proposed to obtain the approximate optimal offloading strategy of dependent tasks in a multi-IIoT device scenario with multiple ECSs. At each decision time t, the scheduling agent observes the state of the IIoT environment and the state of the current decision task to select the offloading action and obtain the reward based on the custom goal. The details are as follows:

[0096] Markov decision process state space: At decision time t, the SDN controller collects the state of the environment and represents it as s t ,s t ∈S. It includes edge server status, device status, and task dependencies. The remaining computing resources, bandwidth, and task waiting queue of the edge server, as well as the computing power of the IIoT device and the currently executed tasks are all necessary information for decision making. As for the dependencies between tasks, since it is a data related to the graph, GCN (graph convolutional neural network) is used to extract its task features, and the result is output as a task-level embedding where D is the embedding dimension of each task and I is the number of tasks generated by IIoT device n.

[0097] At each decision moment, the action space includes the current offloading decision, computing resource allocation, and power size.

[0098] In each decision step, there is a task to be decided. Given a state s t , we can obtain the ETC of all devices in the system at time t. After executing the action, the state changes from s t becomes t+1 , the reward at time step t is defined as the negative increment of the sum of ETC of all users in the IIoT system, calculated as:

[0099]

[0100] Since the MDP proposed in this paper shows high-dimensional and continuous state and action spaces, as well as multi-modal rewards (i.e., delay and energy consumption), we use the SAC algorithm to make unloading decisions. SAC adopts MERL to obtain the optimal strategy that maximizes its expected return and expected entropy, which is defined as:

[0101]

[0102] where E[·] represents the expectation of the random variable, H(π(·|s t )) is the state s t The entropy of policy π in . α is the weight of entropy. The soft-Q value function of policy π is given by According to the Bellman equation, it can be expressed as:

[0103]

[0104] The algorithm proposed in the present invention comprises three components: an evaluation network, a target network and an experience replay pool.

[0105] Then, in the soft policy evaluation phase, we define the y The loss function of the parameter y is:

[0106]

[0107] During the training phase, the network parameters are trained by maximizing the policy’s soft state value function

[0108] J(μ)=E[Q y (s t ,a t )-alogp m (a t |s t )]

[0109] Second, the entropy weight a is adjusted using an adaptive gradient-based method. Specifically, a must be large enough to encourage exploration when the optimal behavior is not determined, and gradually decrease as the optimal behavior is determined. Therefore, the weight a can be updated by minimizing the following objective function:

[0110]

[0111] Finally, to enhance learning stability, the parameters of the target network are updated by slowly tracking the parameters of the evaluation network. and Given by:

[0112]

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A resource allocation method for dependent tasks, characterized by: The method comprises the following steps: S1: Build an industrial Internet system model including cloud servers, edge servers and industrial Internet devices, initialize the number of cloud servers, edge servers and industrial Internet devices in the considered industrial Internet system, and model the dependencies between tasks as a directed acyclic graph; S2: Considering the computational delay of the dependent tasks of industrial Internet devices in local computing, as well as the transmission delay and computational delay in edge servers and cloud servers, a weighted and mathematical model of the total system delay energy consumption of the industrial Internet network is constructed; S3: The problem model is modeled as a Markov decision process, aiming to maximize the cumulative discounted return in the long run; a DRL-based SAC algorithm framework is used to obtain the approximately optimal offloading strategy of dependent tasks in a multi-IIoT device scenario with multiple edge computing servers (ECSs); at each decision time t, the scheduling agent observes the state of the IIoT environment and the state of the current decision task to select the offloading action and obtain the reward based on the custom goal.

2. A method for allocating resources for dependent tasks according to claim 1, characterized in that: In S1, a network system architecture and task dependencies are constructed based on cloud servers, edge servers, and industrial Internet devices, specifically: The applications generated by user n are modeled as a directed acyclic graph DAGG n =(V n ,E n ), which is divided into I interdependent tasks, where V n = {v n,i |i=1,2,...,I} represents the task of the application; It is the dependency relationship between tasks, indicating that the subsequent task must be started after the predecessor task is completed; each task is represented by a tuple (d n,i ,c n,i ) represent the data size of the task and the number of CPU cycles required to complete the task respectively; the start task and the end task are defined as the entry and exit tasks respectively; these two tasks are executed locally.

3. A method for allocating resources for dependent tasks according to claim 2, characterized in that: In S2, according to the characteristics of cloud servers, edge servers and industrial Internet devices, an industrial Internet network communication model, a delay model and an energy consumption model are constructed, specifically: use Represents task v n,i Uninstall strategy; Represents task v n,i Executed on device k, where k = 0 means the task is processed locally, indicates that the task is processed on the edge server, and k=M+1 indicates that the task is processed on the cloud server; The following definitions are proposed: 1) RT n,i :represents task v n,i The preparation time for execution, i.e. the start time; 2) FT n,i :represents task v n,i The completion time during execution, i.e. the moment of completion; Based on the dependencies between computing tasks, computing task v n,i The waiting time is defined as follows: i'∈pred(i) represents task v n,i The set of pre-tasks; IIoT devices complete tasks n,i The total delay is: Compute Task v n,i The actual completion time is given by: FT n,i =RT n,i +t n,i Definition 1 {λ} is an indicator function. If the event λ is true, the function value is 1, otherwise, the value is 0. The calculation task v n,i The energy consumption is: The goal is to find a near-optimal offloading strategy for each application generated by an IIoT device to minimize the average ETC of all users; the ETC of each device is defined as: in, Under the energy consumption and delay constraints of IIoT devices, the offloading strategy, computing resources and power are jointly optimized to minimize the total delay of all device tasks. The scheduling and offloading problem of interdependent computing tasks is formulated as follows: Among them, constraint C1 is the value range of the offloading decision variable; constraint C2 means that a task can only be calculated by one device; constraint C3 represents the value range of the local ECS and collaborative ECS allocation of computing resources; constraint C4 represents the power value range of the IIoT device; constraint C5 calculates the energy consumed by the task.

4. The method for allocating resources for dependent tasks according to claim 2, characterized in that: In S2, after minimizing the weighted sum of the total delay energy consumption of all devices, the optimization problem is modeled as a Markov decision process and solved by the DRL-based SAC algorithm, specifically: Problem P1 is modeled as an MDP with the goal of maximizing the cumulative discounted return in the long run. A DRL-based SAC algorithm framework is used to obtain the approximately optimal offloading strategy for dependent tasks in a multi-IIoT device scenario with multiple ECSs. At each decision time t, the scheduling agent observes the state of the IIoT environment and the state of the current decision task to select an offloading action and obtain a reward based on a custom goal. The details are as follows: Markov decision process state space: At decision time t, the SDN controller collects the state of the environment and represents it as s t ,s t ∈S; the SDN controller collects the state of the environment including edge server state, device state and task dependencies; The remaining computing resources, bandwidth, and task waiting queue of the edge server, as well as the computing power of the IIoT device and the currently executed tasks are all necessary information for decision making. The dependencies between tasks are data related to the graph, and the graph convolutional neural network GCN is used to extract its task features, and the result is output as a task-level embedding. Where D is the embedding dimension of each task, I is the number of tasks generated by IIoT device n; At each decision moment, the action space includes the current offloading decision, computing resource allocation, and power size; In each decision step, there is a task to be decided; given a state s t , obtain the ETC of all devices in the system at time t.

Images