A DRL-based QoE-guaranteed offloading computing system for heterogeneous tasks at the edge of the industrial Internet

Abstract

The present invention discloses a DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system, including several user devices and several edge servers; the present invention comprehensively considers latency, energy consumption, privacy and task heterogeneity, models the QoE-based task offloading problem as a mixed integer nonlinear programming (MINIP) problem, and defines a new QoE indicator consisting of the ratio of actual cost to expected cost, which can effectively quantify the benefits of offloading.

Description

Technical Field

[0001] The present invention relates to the field of the Internet, and in particular to a DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system. Background Art

[0002] The explosive growth of connected devices in the Industrial Internet continues to drive the development of industrial edge computing systems that support the emergence of industrial edge computing systems with high QoS requirements, such as industrial video processing and analysis, industrial production line automatic monitoring and control, industrial robot intelligent control, and industrial augmented reality applications. Typically, these industrial edge computing systems include a variety of complex industrial edge computing tasks, requiring faster and higher-capacity computing resources. At the same time, these applications have diverse and guaranteed Quality of Services (QoS) requirements.

[0003] Due to the limited computing resources and battery power of IoT devices, offloading complex tasks of mobile applications to nearby resource-rich edge servers is becoming a promising solution for ensuring the QoE of heterogeneous tasks at the edge of the Industrial Internet based on DRL. It can achieve faster and more comprehensive data processing and analysis, shorter response time and improved user experience.

[0004] However, effectively scheduling tasks and allocating resources to meet user needs in the IoT combined with edge computing remains a challenging problem. Many existing studies analyze this problem from a QoS perspective, optimizing performance metrics such as minimizing the total latency or energy consumption of tasks, defining task priorities to optimize offloading decisions, and so on. The optimization goal of these approaches is to minimize losses for service providers or users. However, these methods fail to effectively improve user Quality of Experience (QoE). For example, if the user's goal is to save energy, while the model aims to reduce latency, this can lead to offloading decisions that deviate from the user's intent, thereby degrading the user experience.

[0005] In recent years, ensuring user QoE in the IoT, combined with edge computing, has become a highly sought-after issue. However, existing technologies typically construct QoE models based on unified tasks, failing to consider the diverse offloading intentions of tasks in the real world. In actual industrial Internet edge computing environments, tasks generated by different industrial edge computing devices are often heterogeneous. For example, tasks for automated monitoring and control equipment on industrial production lines aim to improve response time, while tasks for industrial video processing and analysis require ensuring privacy during offloading. Summary of the Invention

[0006] The purpose of the present invention is to provide a DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system, including several user devices and several edge servers;

[0007] The user equipment generates task A a (w a ,R a ,c a ,v a ) and upload it to the edge server; a Indicates the task input data size; R a represents the delay constraint, c a Indicates the type of task, v a Indicates the privacy level required for the task;

[0008] The edge server makes task offloading decisions x a =1 indicates task A a Executed on the edge server, x a =0 means task A a Execute locally; decision parameter x a,j =1 means task a is offloaded to the jth edge server, x a,j =0 means that task a is not offloaded to the jth edge server;

[0009] When x a =1, the edge server executes task A a , when x a = 0, the edge server transmits the execution policy to the user device, so that the user device executes task A a .

[0010] The tasks generated by users include category C; some of the tasks in category C are delay-sensitive tasks, while the other tasks are delay-insensitive tasks.

[0011] The uplink spectrum of the edge server is divided into several sub-channels, each with a bandwidth of ω. The user selects a sub-channel to upload the task to the edge server.

[0012] The edge server stores a task delay model;

[0013] The task delay model is used to calculate the total delay t spent by the task a ,Right now:

[0014]

[0015] Where, the computing time of the task on the edge server is λ is the scaling factor; Assign task A to user i a computing power; The local computing time of the task; Assign edge server j to task A a computing power;

[0016] Among them, Task A a Transmission delay As shown below:

[0017]

[0018] The uplink data transmission rate r between local device i and edge server j i,j As shown below:

[0019]

[0020] Where h i,j is the channel gain between local device i and edge server j, σ 2 represents the power of background noise; p i is the transmission power of local device i; ω is the bandwidth of the uplink subchannel;

[0021] The constraints of the task delay model include the computing resource constraints of the user device and the computing resource constraints of the edge server;

[0022] The computing resource constraints of user devices are as follows:

[0023]

[0024] Where, F i l is the computing power of the i-th user;

[0025] The computing resource constraints of the edge server are as follows:

[0026]

[0027] Where, is the computing power of the jth edge server; Assign edge server j to task A a computing power;

[0028] The edge server stores an energy consumption model;

[0029] The energy consumption model is used to calculate the total energy consumption of the task e a ,Right now:

[0030]

[0031] In the formula, transmission energy consumption p i,a Transfer task A to local device i a transmission power;

[0032] Among them, Task A a Local computing energy consumption As shown below:

[0033]

[0034] Where κ is the effective switch capacitance.

[0035] The edge server stores a decision optimization model; the steps of establishing the decision optimization model include:

[0036] 1) Calculate and complete task A a Completion computing resources Right now:

[0037]

[0038] Where, ρ is the statistical percentage value; R' a (ρ) represents the statistical delay constraint of task a when counting ρ percent of historical tasks;

[0039] 2) Computational Task A a Computing resources that need to be obtained from local devices and computing resources required from edge servers Right now:

[0040]

[0041] 3) Establish energy consumption constraints (10) and privacy constraints (11), namely:

[0042]

[0043] Where, Indicates the actual privacy level of the task;

[0044] 4) Computational Task A a QoE value a ,Right now:

[0045]

[0046] Where, These three variables are related to c a The weight parameters of different types of tasks represent the different levels of attention paid by the current task to the three indicators of latency, energy consumption, and privacy; g is a preset constant; the values of the weight parameters for different types of tasks are shown in Table II;

[0047] 5) Establish the optimization objectives and constraints of the decision optimization model, namely:

[0048]

[0049] C2:(4),(5),

[0050]

[0051] C4:(10),(11)

[0052] Where C1 represents the task execution strategy; C1 = 0 means the task is executed locally; C1 = 1 means the task is offloaded to the edge server for execution; C2 is the allocated computing resource constraint; C3 is the task delay constraint; C4 is the energy and privacy constraint of heterogeneous tasks; R a,t Represents a delay constraint.

[0053] The edge server will task A a Input into the decision optimization model to obtain the execution strategy. The steps include:

[0054] 1) Get the current state vector U t is the state vector of the local device, E t is the edge server state vector, CH t is the channel gain information vector;

[0055] 2) The state vector s t As input, the edge server selects the action ac through the binary action mask method and the ∈-greedy method t If ac t =0, the task is executed on the user device, ac t =I, the task is offloaded to the edge server numbered I for execution; I∈N; N is the number of edge servers;

[0056] 3) Action ac t The execution of will change the current environment state. The environment change parameters include the state of the local device U t , Edge Server Status E t And the parameters related to the new incoming tasks The edge server observes the new state vector s t+1 , and get the reward r after the current action act is completed t =QoE t ;QoE t Indicates the QoE value;

[0057] 4) The state vector s i and the state vector s i+1 Input into the decision optimization model to obtain parameters and parameters Representative action ac i The estimated Q value is, is a vector, each value in the vector represents an estimated Q value of a behavior; θ and θ - are the current parameters of the online network and the target network, respectively;

[0058] Calculate the state vector s t Next action ac i The expected Q value of Right now;

[0059]

[0060] Where γ is the discount factor between current rewards and future rewards. represents the target network estimated s i+1 The value vector of all actions under r i Indicates reward;

[0061] Calculate the loss L i (θ), that is:

[0062]

[0063] Where bs is the number of state vectors;

[0064] 5) Based on L i The gradient of (θ) is back-propagated to update θ. When the number of updates of θ meets the update frequency, the target network parameter θ is - Replace with θ; θ is a hyperparameter of the reinforcement learning algorithm, which is generally set based on experience and is described in the delay section (the parameter update frequency of the target network is 1000). Here θ - These are the parameters of the target network.

[0065] 6) Calculate the parameter Q(s) i ),Right now:

[0066]

[0067] Where, Q(s i ) is a one-dimensional vector, where each value represents the Q value estimated by D3QN for a decision in the current state; Λ(s i ) is the prediction of the expected reward for all actions, V(si) is the evaluation value of state si; |Λ| represents the absolute value of the expected reward prediction for the action;

[0068] 7) Calculate the decision action ac i ,Right now:

[0069]

[0070] Where τ is a random number from 0 to 1; ∈ is the exploration factor; ac i =j means that the task calculated for the current environment is offloaded to the edge server j. i Represents the action obtained by reinforcement learning for the current environment, which is one of the four elements of reinforcement learning Markov problem modeling. i = j means that the task calculated for the current environment is offloaded to the edge server j, and at this time x a,j = 1. The binary mask vector m(s i ) is equal to the size of the action space, m(s i ) are initialized to 1. However, if the available resources of the local device or edge server are scarce, m(s i ) is set to 0.

[0071] The technical effect of the present invention is unquestionable. The present invention defines a new QoE (Quality of Experience) indicator and adopts a statistical strategy to relax the strict constraints of heterogeneous tasks, thereby improving the success rate of task offloading. In addition, the present invention models the QoE-guaranteed heterogeneous task offloading problem as a mixed-integer nonlinear programming (MINIP) problem. Traditional numerical optimization methods are inefficient in solving such problems. Therefore, this embodiment applies a deep reinforcement learning (DRL) algorithm to make the best offloading decision that meets the task offloading intention. Experimental results show that the algorithm of this embodiment effectively guarantees the global QoE performance and improves the offloading success rate of heterogeneous tasks.

[0072] This paper proposes a DRL-based QoE-guaranteed offloading computation system for heterogeneous tasks at the edge of the Industrial Internet, used to make optimal offloading decisions. Taking into account latency, energy consumption, privacy, and task heterogeneity, this paper models the QoE-based task offloading problem as a mixed-integer nonlinear programming (MINIP) problem and defines a new QoE metric consisting of the ratio of actual cost to expected cost, which can effectively quantify the benefits of offloading.

[0073] To ensure QoE, this paper employs a DRL-based heterogeneous task offloading algorithm. This algorithm has two components: 1) accurately estimating the benefits of decisions; and 2) formulating rules for heterogeneous tasks and then filtering out unreasonable decisions using a binary action masking approach. Furthermore, this embodiment proposes a DRL-based statistical strategy for offloading computing systems to ensure QoE for heterogeneous tasks at the edge of the Industrial Internet, improving task success rates.

[0074] The present invention simulates a real edge IoT environment and conducts extensive experiments to demonstrate the performance of the model of this embodiment in improving global QoE and task success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Offloading tasks in edge IoT environments;

[0076] Figure 2 DuelingDoubleDeepQ-Learning (D3QN) algorithm process;

[0077] Figure 3 It is the D3QN network structure;

[0078] Figure 4 is the QoE mean;

[0079] Figure 5 The success rate of task execution;

[0080] Figure 6 is the task execution success rate when ρ takes different values. DETAILED DESCRIPTION

[0081] The present invention will be further described below with reference to the following examples, but it should not be understood that the scope of the present invention is limited to the following examples. Without departing from the above technical ideas of the present invention, various substitutions and modifications can be made according to common technical knowledge and customary means in the art, and all should be included in the scope of protection of the present invention.

[0082] Example 1:

[0083] See also Figures 1 to 6 A DRL (Deep Reinforcement Learning)-based QoE (Quality of Experience)-ensured offload computing system for heterogeneous tasks at the edge of the Industrial Internet, including several user devices and several edge servers.

[0084] The user equipment generates task A a (w a ,R a ,c a ,v a ) and upload it to the edge server; a Indicates the task input data size; R a represents the delay constraint, c a Indicates the type of task, v a Indicates the privacy level required for the task;

[0085] The edge server makes task offloading decisions x a =1 indicates task A a Executed on the edge server, x a =0 means task A a Execute locally; decision parameter x a,j =1 means task a is offloaded to the jth edge server, x a,j =0 means that task a is not offloaded to the jth edge server;

[0086] When x a =1, the edge server executes task A a , when x a = 0, the edge server transmits the execution policy to the user device, so that the user device executes task A a .

[0087] User-generated tasks fall into C categories; some of these tasks are delay-sensitive, while others are delay-insensitive. The edge server's uplink spectrum is divided into several subchannels, each with a bandwidth of ω. Users select a subchannel to upload their tasks to the edge server. The edge server stores a task delay model.

[0088] The task delay model is used to calculate the total delay t spent by the task a ,Right now:

[0089]

[0090] Where, the computing time of the task on the edge server is The computing power allocated to task a for user i; The local computing time of the task; Assign edge server j to task A a The computing power of λ is the proportional factor;

[0091] Among them, Task A a Transmission delay As shown below:

[0092]

[0093] The uplink data transmission rate r between local device i and edge server j i,j As shown below:

[0094]

[0095] Where h i,j is the channel gain between local device i and edge server j, σ 2represents the power of background noise; p i is the transmission power of local device i; ω is the bandwidth of the uplink channel;

[0096] The constraints of the task delay model include the computing resource constraints of the user device and the computing resource constraints of the edge server;

[0097] The computing resource constraints of user devices are as follows:

[0098]

[0099] Where, F i l is the computing power of the i-th user;

[0100] The computing resource constraints of the edge server are as follows:

[0101]

[0102] Where, is the computing power of the jth edge server; Assign edge server j to task A a computing power;

[0103] The edge server stores an energy consumption model;

[0104] The energy consumption model is used to calculate the total energy consumption of the task e a ,Right now:

[0105]

[0106] In the formula, transmission energy consumption p i,a Transfer task A to local device i a transmission power;

[0107] Among them, Task A a Local computing energy consumption As shown below:

[0108]

[0109] Where κ is the effective switch capacitance which depends on the hardware architecture.

[0110] The edge server stores a decision optimization model; the steps of establishing the decision optimization model include:

[0111] 1) Calculate and complete task A a Completion computing resources Right now:

[0112]

[0113] Where, ρ is the statistical percentage value; R' a (ρ) represents the statistical delay constraint of task a when counting ρ percent of historical tasks;

[0114] 2) Computational Task A a Computing resources that need to be obtained from local devices and computing resources required from edge servers Right now:

[0115]

[0116] 3) Establish energy consumption constraints (10) and privacy constraints (11), namely:

[0117]

[0118] Where, Indicates the actual privacy level of the task; v max It is the highest level of privacy setting; Indicates the privacy level of the edge server;

[0119] 4) Computational Task A a QoE value a ,Right now:

[0120]

[0121] Where, These three variables are related to c a The weight parameters of different types of tasks represent the different levels of attention paid by the current task to the three indicators of latency, energy consumption, and privacy. g is a preset constant. The values of the weight parameters for different types of tasks are shown in Table II. tasksuccess indicates that the task can be successfully executed, and taskfail indicates that the task is marked as failed.

[0122] 5) Establish the optimization objectives and constraints of the decision optimization model, namely:

[0123]

[0124] C2:(4),(5),

[0125]

[0126] C4:(10),(11)

[0127] Where C1 represents the task execution strategy; C1 = 0 means the task is executed locally; C1 = 1 means the task is offloaded to the edge server for execution; C2 is the allocated computing resource constraint; C3 is the task delay constraint; C4 is the energy and privacy constraint of heterogeneous tasks; R a,t Represents a delay constraint.

[0128] The edge server will task A a Input into the decision optimization model to obtain the execution strategy. The steps include:

[0129] 1) Get the current state vector U t is the state vector of the local device, E t is the edge server state vector, CH t is the channel gain information vector;

[0130] 2) The state vector s t As input, the edge server selects the action ac through the binary action mask method and the ∈-greedy method t If ac t =0, the task is executed on the user device, ac t =I, the task is offloaded to the edge server numbered I for execution; I∈N; N is the number of edge servers;

[0131] 3) Action ac t The execution of will change the current environment state. The environment change parameters include the state of the local device U t , Edge Server Status E t And the parameters related to the new incoming tasks The edge server observes the new state vector s t+1 , and get the reward r after the current action act is completed t =QoE t ;QoE t Indicates the QoE value;

[0132] 4) The state vector s i and the state vector s i+1 Input into the decision optimization model to obtain parameters and parameters Representative action ac i The estimated Q value is, is a vector, each value in the vector represents an estimated Q value of a behavior; θ and θ - are the current parameters of the online network and the target network, respectively;

[0133] Calculate the state vector s t Next action ac i The expected Q value of Right now;

[0134]

[0135] Where γ is the discount factor between current rewards and future rewards. represents the target network estimated s i+1 The value vector of all actions under r i Indicates reward;

[0136] Calculate the loss L i (θ), that is:

[0137]

[0138] Where bs is the number of state vectors;

[0139] 5) Based on L i The gradient of (θ) is back-propagated to update θ. When the number of updates of θ meets the update frequency, the target network parameter θ is - Replace with θ; θ is a hyperparameter of the reinforcement learning algorithm, which is generally set based on experience and is described in the delay section (the parameter update frequency of the target network is 1000). Here θ - These are the parameters of the target network.

[0140] 6) Calculate the parameter Q(s) i ),Right now:

[0141]

[0142] Where, Q(s i ) is a one-dimensional vector, where each value represents the Q value estimated by D3QN for a decision in the current state; Λ(s i ) is the prediction of the expected reward for all actions, V(si) is the evaluation value of state si; |Λ| represents the absolute value of the expected reward prediction for the action;

[0143] 7) Use the ∈-greedy (greedy strategy) method to calculate the decision action ac i ,Right now:

[0144]

[0145] Where τ is a random number from 0 to 1; ∈ is the exploration factor; ac i =j means that the task calculated for the current environment is offloaded to the edge server j. i Represents the action obtained by reinforcement learning for the current environment, which is one of the four elements of reinforcement learning Markov problem modeling. i = j means that the task calculated for the current environment is offloaded to the edge server j, and at this time xa,j = 1. The binary mask vector m(s i ) is equal to the size of the action space, m(s i ) are initialized to 1. However, if the available resources of the local device or edge server are scarce, m(s i ) is set to 0. randomaction represents a random number;

[0146] Example 2:

[0147] See also Figures 1 to 6 ,A DRL-based industrial Internet edge heterogeneous task QoE assured offloading computing system includes several user devices and several edge servers;

[0148] Support edge IoT environments such as Figure 1 As shown. The edge server deployed in the base station can provide computing power for users. Users can offload tasks to the edge server for a better experience. The tasks generated by IoT users can be divided into C types according to different intentions. In this article, this embodiment sets C=3. It is assumed that the first type of task is delay-aware, the second type of task is energy-aware, and the third type of task is privacy-aware. The latter two types of tasks are also considered to be delay-insensitive tasks. There are M users and N edge servers in the system. Users can establish a connection with the edge server through a wireless network. Since the task execution result data is usually small, the transmission consumption only considers the uplink energy consumption and time consumption, while ignoring the return overhead. In order to improve efficiency, this embodiment considers an orthogonal frequency division multiple access scheme, and the uplink spectrum of edge computing is divided into several equal sub-channels, and the bandwidth of each sub-channel is ω. A task can only select one sub-channel to transmit data. The feature of a task model is a tuple consisting of four parameters, represented by A a (w a ,R a ,c a ,v a ) indicates that A a Represents tasks a, w a Indicates the task input data size; R a represents the delay constraint, c a Indicates the type of task, v a Indicates the privacy level required for a task. This embodiment assumes that tasks are atomic and cannot be divided into subtasks. Some symbols used in this paper are listed in Table I.

[0149] Table I

[0150]

[0151]

[0152] B.HeterogeneousTaskOffloadingModel (Heterogeneous Task Offloading Model)

[0153] This example defines a binary variable x a,j To represent the task offloading decision, where x a,j =1 means task a is offloaded to the jth edge server, otherwise x a,j = 0. Therefore, let where x a =1 means the task is executed on the edge server, otherwise the task is executed locally. Each edge server has a privacy level, which is expressed as v j This embodiment assumes that local devices always maintain maximum privacy for their tasks. Task offloading primarily consists of three phases: 1) uploading task data via a wireless channel, 2) executing the task on the edge server, and 3) downloading the results from the edge server. This embodiment focuses on the first two phases and proposes a two-phase task delay model and energy consumption model as follows.

[0154] a) Task delay model: define F i l and is the computing power of the i-th user and the j-th edge server. a = 0, the computing power allocated to task a by user i is Therefore, the constraints on local computing resources are:

[0155]

[0156] The local computation time t of task a a It can be calculated as follows:

[0157] The number of CPU cycles required to complete a task is proportional to the size of the task data, with a scaling factor of λ. Otherwise, when x a =1, let x a,j =1, the computing power allocated to task a by edge server j is The corresponding constraints are:

[0158]

[0159] Computation time of the task at the edge It can be determined by the following formula:

[0160] Device i has transmission power p i , the uplink data transmission rate r between local device i and edge server ji,j It can be calculated as follows:

[0161]

[0162] where h i,j is the channel gain between local device i and edge server j, σ 2 represents the power of background noise. Therefore, the transmission delay of task a can be calculated as:

[0163]

[0164] Therefore, the total delay ta taken by the task is as follows:

[0165]

[0166] b) Energy Consumption Model: Energy consumption is primarily composed of local computation energy and transmission energy when uploading data to the edge server. Because this embodiment primarily considers offloading decisions from the user's perspective, the energy consumption of computation on the edge server is ignored. The local computation energy consumption of task a can be calculated as:

[0167]

[0168] Where κ is the effective switching capacitance that depends on the hardware architecture. The transmission energy consumption is given by: Therefore, the total energy consumption of the task is:

[0169]

[0170] c) Statistical delay bound for delay-insensitive tasks: Statistical delay bound has been proposed and proven to be a powerful wireless transmission technology. However, it is rarely studied in edge task offloading research. First, this embodiment defines a statistical percentage value ρ and lets R' a (ρ) represents the statistical delay constraint of task a when counting ρ percent of historical tasks. R' a (ρ) can be calculated by the inverse cumulative distribution function of the historical tasks, which means

[0171]

[0172] If the current edge environment has sufficient resources, then R' a (ρ) will be close to R a Otherwise R' a (ρ) will relax R according to the scarcity of resources a The effectiveness of the statistical delay-constrained model in improving the success rate of task execution without significantly affecting user experience.

[0173] C.QoEOptimizationModel (QoE optimization model)

[0174] First, this embodiment defines several execution failure scenarios for heterogeneous tasks. Indicates the computing resources required for task a to be completed before the deadline on the local device and the edge server. It can be calculated as follows:

[0175]

[0176] If task a cannot be obtained from the local device Computing resources or obtaining j(j∈N) computing resources from the edge server, task a fails. Especially for energy-aware tasks, they can only be offloaded to the edge server when the energy consumption of transmission is less than the energy consumption of local computing, that is:

[0177]

[0178] For privacy-aware tasks, they can only be executed on the edge server if the edge server’s privacy level is higher than the task’s privacy requirement. The actual privacy level of the task can be expressed as represents the actual privacy level of the task; where v max is the highest level of privacy setting. Therefore, the privacy constraints are:

[0179]

[0180] If there is no feasible execution decision that can meet the energy consumption requirements or privacy requirements of the task, the task is marked as failed. For failed tasks, their QoE values are set to g. Finally, the QoE values of task a are as follows:

[0181]

[0182] in and is the weight parameter of these three variables for CA type tasks. For a successful task, its QoE value is composed of the degree of optimization of the model in terms of delay, energy consumption and privacy. Different types of tasks have different degrees of attention to the optimization of delay, energy consumption and privacy. According to the above formula, Will reflect the heterogeneity between tasks.

[0183] Therefore, the final optimization goal is:

[0184]

[0185] C2:(4),(5),

[0186]

[0187] C4:(10),(11)

[0188] Where C1 indicates that the task can only be executed locally or offloaded to the edge server. C2 declares that the allocated computing resources cannot exceed the total amount of the device or server. C3 represents the delay constraint of the task. C4 specifies the energy and privacy constraints of heterogeneous tasks. It can be seen that xa is a discrete integer variable that is coupled with the continuous variable. Therefore, the offloading problem is defined as a MINIP problem, which has been proved to be an NP-hard problem. This embodiment uses DRL to solve the decision optimization problem. In addition, this embodiment applies a binary mask method to the action to filter out actions that will cause the task to fail.

[0189] Reinforcement learning algorithms based on Q-values have proven effective for discrete action problems. In the algorithm proposed in this example, DuelingDoubleDeepQ-Learning (D3QN) is used to estimate the benefit of each decision. Compared to naive deep Q-learning algorithms, D3QN optimizes its Q-value calculation method and network architecture, addressing the overfitting problem in calculating the Q-value of the Bellman equation and eliminating irrelevant states for actions. Specifically, the Q-value estimates the benefit of an offloading decision.

[0190] The unloading process based on reinforcement learning can be regarded as a Markov reward process, using the four-tuple <s t ,ac t ,s t+1 ,r t > represents, respectively, the state, action, next state, and reward. According to the problem of this embodiment, they are defined as follows:

[0191] Status t :s t is a vector representing the current state of the environment, i.e. U t is the state vector of the local device, E t is the edge server state vector, CH t is the channel gain information vector.

[0192] Action ac t :ac t Represents state s t The decision below. If ac t = 0, the task is executed locally, if ac t =I(I∈N), the task is offloaded to the edge server numbered I.

[0193] Next state s t+1 :When the agent is in s t Execute ac i When the environment changes, the agent observes a new state s t+1 .

[0194] Reward t :After the task is completed, a QoE value calculated by (12) will be fed back. In this embodiment, it is set to r t Therefore, r t Equal to QoE t , and is used to judge ac t of income.

[0195] The process of the D3QN algorithm is as follows Figure 2 As shown. First, the agent obtains the state s according to the current environment t , and s t As the input of the online network. The online network selects the action ac by adopting the binary action mask method and ∈-greedy algorithm t The agent performs action ac t And get the reward rt. After that, update the environment and the agent will observe a new state s t+1 Based on the above, s t ,ac t s t+1 , and r t A new experience is formed and appended to the experience replay pool. When the number of experiences in the pool is greater than the batch size bs, bs number of experiences are sampled to train and update the parameters of the online network. D3QN uses the ∈-greedy strategy for exploration and experience replay for exploitation, ultimately capturing the reward mapping from state to action.

[0196] a) UpdateNetworkParameters: Randomly sample bs experience, assuming one of them is <s i ,ac i ,s i+1 ,r i >. Let ac i represents a fixed action, and ac represents all possible actions. i and s i+1 Enter the online network and it will return and The former represents the action ac i The estimated Q value of the value, which is a vector, each value in the vector represents an estimated Q value of a behavior. At the same time, s i+1 Send it to the target network and get That is, the target network estimates s i+1 The value vector of all actions under θ and θ - are the current parameters of the online network and the target network respectively. Therefore, this embodiment can calculate s i Next action ac i The expected Q value of is as follows:

[0197]

[0198] Where γ is the discount factor between current rewards and future rewards. It can be seen that the expected Q value is composed of current and future rewards.

[0199] Loss L i (θ) is the mean squared error (MSE) between the estimated Q-value and the expected Q-value, which can be calculated as follows:

[0200]

[0201] Finally, by using L i The gradient of (θ) is back-propagated to update θ. After θ is updated for a specific round, θ - Replace with θ.

[0202] DuelingNetworkArchitecture (target network architecture): The structure of the online network and the target network is as follows Figure 3 As shown in Figure 2. The structure consists of four layers: input layer, hidden layer, dual layer and output layer. The hidden layer consists of multiple fully connected layers. The dual layer output vector Λ(s i ) and value V(s i ). Λ(s i ) is the prediction of the expected reward for all actions, V(s i ) is the state s i Finally, the output of the output layer is:

[0203]

[0204] Where Q(s i ) is a one-dimensional vector, where each value represents the Q value estimated by D3QN for a decision in the current state.

[0205] B.BinaryActionMaskforHeterogeneousTask (binary action mask for heterogeneous tasks)

[0206] Once the rewards of all actions are obtained, this embodiment can select the action with the largest Q value as the scheduling decision for the current task. However, applying some actions will cause the task to fail. In order to eliminate those inappropriate actions, we define a binary mask vector m(si ). m(s i ) is equal to the size of the action space, m(s i ) are initialized to 1. However, if the available resources of the local device or edge server are scarce, m(s i ) is set to 0. For example, if there are four edge servers, then m(s i )=[1,1,1,1,1], where the first value represents the local device and the rest represent the edge servers. If the available resources of the local device are less than And the available resources of edge servers numbered 2 and 4 are less than Then m(s i )=[0,1,0,1,0]. For energy-aware tasks, if the transmission energy consumption is greater than the local computing energy consumption, the mask value of the corresponding edge server is set to 0; for privacy-aware tasks, if the privacy level of the edge server is lower than the privacy requirement of the task, the corresponding mask value is set to 0. Therefore, according to the ∈-greedy strategy, this embodiment can make the scheduling decision for the current task:

[0207]

[0208] τ is a random number from 0 to 1 generated at each step, and ∈ is the exploration factor. Reasonable adjustment of ∈ can allow D3QN to explore useful experiences faster.

[0209] Example 3:

[0210] An experiment on ensuring the QoE of heterogeneous tasks at the edge of the industrial Internet based on DRL to offload the computing system. This embodiment demonstrates the evaluation results of the proposed QoE model and the performance of the heterogeneous task offloading algorithm based on DRL through simulation.

[0211] A. Experimental Environment

[0212] This embodiment considers a scenario with multiple edge servers and multiple IoT devices, with the devices randomly distributed around the edge servers. The experimental parameters are set reasonably and are based on actual applications. The number of each IoT device is set to 20, and the number of edge servers is set to 10. Therefore, there are a total of 60 IoT devices. Each device can generate a task in a time slot with a probability of 50%. The agent collects tasks and makes offloading decisions every five time slots. The initial value of the statistical percentage value ρ is set to 95%. One time slot is equal to 1ms. The channel gain h i,j Equal to 10 -5 . Failed mission

[0213] The QoE value of g is set to -2. The main simulation parameters of the QoE model are shown in Table II.

[0214] Table II

[0215]

[0216]

[0217] The D3QN algorithm proposed in this example is implemented in the PyTorch framework. For experimental comparison, this example provides three other baseline algorithms. The detailed structure of these four algorithms is as follows:

[0218] D3QN: The number of neurons in the two hidden layers is set to 256 and 128 respectively. The learning rate is set to 10 -5 , the discount factor γ is set to 0.95. The update frequency of the target network is set to 1000 rounds, and the value of the exploration factor ∈ is set to 0.95. The maximum capacity of the experience replay pool is set to 10 5 In this embodiment, the batch size bs is set to 128.

[0219] DQN: The calculation method of Q value in DQN is:

[0220]

[0221] (18) DQN does not use the dual network architecture. Its hidden layer consists of four fully connected layers, with the number of neurons in each layer being 256, 128, 64, and 32, respectively. The rest of the parameters and structure of DQN refer to D3QN.

[0222] Greedy method: The Greedy method always chooses the action with the highest reward among the available actions.

[0223] Random method: The Random method randomly selects an available operation as the task offloading decision.

[0224] B. Experimental Results

[0225] This example first discusses the performance of D3QN and three other baseline algorithms in the proposed QoE model. This example primarily measures performance based on two metrics: average QoE and task success rate. The four algorithms were evaluated over 12,000 time slots, all applying a statistical relaxation strategy. Figure 4 The fluctuation of the average QoE value over time is illustrated. It can be clearly seen that the DRL algorithm has a significant advantage over other algorithms in terms of QoE guarantee. Compared with the baseline algorithm, D3QN improves the QoE performance by approximately 25%, 13%, and 6%, respectively. Figure 5We plotted the task success rate over time, and found that D3QN outperformed the other algorithms and increased the task success rate to approximately 95.5%. The reason for D3QN’s superior performance is that it focuses on maximizing the overall reward and estimates the Q-value of actions more accurately than DQN.

[0226] Then, this embodiment conducts an experiment to verify the influence of the statistical percentage value ρ on the task success rate. Figure 6 This section illustrates the task success rates of D3QN under different statistical delay relaxation strategies. Specifically, ρ = 0 indicates that the statistical delay relaxation strategy is not applied. This example uses a polynomial fitting method to smooth the curves for a clear comparison. This shows that statistical delay constraints have a positive effect on improving task success rates. Larger values of ρ indicate less sensitivity to delay.

[0227] The greater the task delay relaxation, the more historical tasks need to be covered to calculate the statistical delay. When ρ is 95% and 99%, as T increases, their task success rate curves overlap because the statistical delays obtained by the two strategies are similar after the number of historical tasks becomes larger.

[0228] In this embodiment of the present invention, we studied offloading strategies for heterogeneous tasks. To improve the success rate of tasks, we proposed applying statistical delay constraints to relax the deterministic delay constraints of delay-insensitive tasks. Subsequently, we defined a novel QoE metric to accurately measure the benefits of task scheduling on user experience. To maximize global QoE and task success rate, we designed a DRL-based algorithm to optimize decision making. Finally, experimental results verified that D3QN outperformed other baseline algorithms and demonstrated the positive effect of statistical delay constraints on ensuring task success rate.

Claims

1. A DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system, characterized by: Includes several user devices and several edge servers; The user equipment generates task A a (w a ,R a ,c a ,v a ) and upload it to the edge server; a Indicates the task input data size; R a represents the delay constraint, c a Indicates the type of task, v a Indicates the privacy level required for the task; The edge server makes task offloading decisions x a =1 indicates task A a Executed on the edge server, x a =0 means task A a Execute locally; decision parameter x a,j =1 means task a is offloaded to the jth edge server, x a,j =0 means that task a is not offloaded to the jth edge server; When x a =1, the edge server executes task A a , when x a = 0, the edge server transmits the execution policy to the user device, so that the user device executes task A a ; The edge server will task A a Input into the decision optimization model to obtain the execution strategy. The steps include: 1) Get the current state vector U t is the state vector of the local device, E t is the edge server state vector, CH t is the channel gain information vector; These three variables are related to c a The weight parameters of the task type represent the different levels of attention paid by the current task to the three indicators of latency, energy consumption, and privacy; 2) The state vector s t As input, the edge server selects the action ac through the binary action mask method and the ∈-greedy method t If ac t =0, the task is executed on the user device, ac t =I, the task is offloaded to the edge server numbered I for execution; I∈N; N is the number of edge servers; 3) Action ac t The execution of will change the current environment state. The environment change parameters include the state of the local device U t , Edge Server Status E t And the parameters related to the new incoming tasks The edge server observes the new state vector s t+1 , and get the reward r after the current action act is completed t =QoE t ;QoE t Indicates the QoE value; 4) The state vector s i and the state vector s i+1 Input into the decision optimization model to obtain parameters and parameters And calculate the state vector s t Next action ac i The expected Q value of Right now; Where γ is the discount factor between current rewards and future rewards; represents the target network estimated s i+1 The value vector of all actions under r i Indicates reward; Calculate the loss L i (θ), that is: Where bs is the number of state vectors; 5) Based on L i The gradient of (θ) is back-propagated to update θ. When the number of updates of θ meets the update frequency, the target network parameter θ is - Replace with θ; 6) Calculate the parameter Q(s) i ),Right now: Where, Q(s i ) is a one-dimensional vector, where each value represents the Q value estimated by D3QN for a decision in the current state; Λ(s i ) is the prediction of the expected reward for all actions, V(s i ) is the state s i The evaluation value of |Λ| represents the absolute value of the expected reward prediction of the action; 7) Calculate the decision action ac i ,Right now: Where τ is a random number from 0 to 1; ∈ is the exploration factor; ac i = j means that the task calculated for the current environment is offloaded to the edge server j; m(s i ) is a binary mask vector.

2. The DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system according to claim 1 is characterized by: The tasks generated by users include category C; some of the tasks in category C are delay-sensitive tasks, while the other tasks are delay-insensitive tasks.

3. The DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system according to claim 1 is characterized by: The uplink spectrum of the edge server is divided into several sub-channels, each with a bandwidth of ω. The user selects a sub-channel to upload the task to the edge server.

4. The DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system according to claim 1 is characterized by: The edge server stores a task delay model; The task delay model is used to calculate the total delay t spent by the task a ,Right now: Where, the computing time of the task on the edge server is The local computing time of the task; Assign edge server j to task A a The computing power of λ is the proportional factor; Among them, Task A a Transmission delay As shown below: The uplink data transmission rate r between local device i and edge server j i,j As shown below: Where h i,j is the channel gain between local device i and edge server j, σ 2 represents the power of background noise; p i is the transmission power of local device i; ω is the bandwidth of the uplink subchannel; The constraints of the task delay model include the computing resource constraints of the user device and the computing resource constraints of the edge server; The computing resource constraints of user devices are as follows: Where, F i l is the computing power of the i-th user; Assign task A to user i a computing power; The computing resource constraints of the edge server are as follows: Where, is the computing capacity of the jth edge server.

5. The DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system according to claim 1 is characterized by: The edge server stores an energy consumption model; The energy consumption model is used to calculate the total energy consumption of the task e a ,Right now: Where, is the transmission energy consumption; Among them, Task A a Local computing energy consumption As shown below: Where κ is the effective switch capacitance that depends on the hardware architecture; λ is the scaling factor; Assign task A to user i a computing power.

6. The DRL-based industrial Internet edge heterogeneous task QoE guaranteed offloading computing system according to claim 1 is characterized by: The edge server stores a decision optimization model; the steps of establishing the decision optimization model include: 1) Calculate and complete task A a Completion computing resources Right now: Where, ρ is the statistical percentage value; R' a (ρ) represents the statistical delay constraint of task a when counting ρ percent of historical tasks; 2) Computational Task A a Computing resources that need to be obtained from local devices and computing resources required from edge servers Right now: Where λ is the proportional factor; For Task A a transmission delay; 3) Establish energy consumption constraints (10) and privacy constraints (11), namely: Where, Indicates the actual privacy level of the task; Represents Task A a local computing energy consumption; 4) Computational Task A a QoE value a ,Right now: Where, These three variables are related to c a The weight parameters of the type task represent the different levels of attention paid by the current task to the three indicators of latency, energy consumption, and privacy; g is a constant; 5) Establish the optimization objectives and constraints of the decision optimization model, namely: s.t.C1: C2:(4),(5), C3: C4:(10),(11) Where C1 represents the task execution strategy; C1 = 0 means the task is executed locally; C1 = 1 means the task is offloaded to the edge server for execution; C2 is the allocated computing resource constraint; C3 is the task delay constraint; C4 is the energy and privacy constraint of heterogeneous tasks; R a,t Represents a delay constraint.

Images