D3qn-based edge-enabled iiot online computation migration method

Abstract

The application discloses an edge-enabled IIoT online computing migration method based on D3QN, and relates to the technical field of computers, and comprises the following steps: abstracting a current industrial environment into an edge-enabled IIoT network model, and modeling tasks in the IIoT to obtain a multi-constraint task model; establishing a multi-constraint task computing offloading problem model including time delay and energy consumption according to the power of a local device, the channel transmission bandwidth and the computing capacity of a server; designing a D3QN network based on the multi-constraint task model and the multi-constraint task computing offloading problem model, processing IIoT task online computing migration problems into a Markov decision process, including a state State, an action Action and a reward Reward; adding noise in the D3QN network, establishing a deep reinforcement learning-based algorithm Noisy D3QN according to the model, and obtaining an optimal task offloading decision through solving. The application considers the large scale and high complexity of industrial data, adopts the Noisy D3QN algorithm to obtain an optimal migration strategy, and efficiently optimizes system time delay and energy consumption.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a D3QN-based edge-enabled IIoT online computing migration method. Background Art

[0002] With the introduction and reform of Industry 4.0, the Industrial Internet of Things (IIoT) has become a key application of the Internet of Things in industrial scenarios. By continuously integrating technologies such as mobile communications and intelligent analytics into every aspect of industrial production, the IIoT significantly improves manufacturing efficiency, reduces product costs and resource consumption, and improves product quality. Ultimately, it promotes the intelligent transformation of traditional industries and promotes sustainable development. In the future, the IIoT will develop towards intelligent production, networked collaboration, personalized customization, and service-oriented transformation.

[0003] However, the growing complexity of infrastructure and the strong demand for high-performance computing are challenging the comprehensive level of IIoT technology. Due to the high demands of real-time performance and collaborative work in the industrial sector, most IIoT devices and sensors are limited by their computing resources and battery capacity, making it difficult to support the growing number of compute-intensive and latency-sensitive applications. Furthermore, within the extremely fragmented application scenarios of IIoT, the types of tasks are also extremely diverse, and building IIoT requires addressing the heterogeneity of data.

[0004] To address this, Multi-access Edge Computing (MEC), a new service computing model, has emerged as an effective solution to these issues. As a key 5G technology, MEC reduces latency by offloading computing tasks from IIoT devices to nearby, resource-rich edge servers. This allows IIoT systems to directly leverage the abundant computing resources and storage capacity of edge servers, effectively overcoming the high data communication latency caused by physical distance in traditional cloud computing. To address the heterogeneity of IIoT data, MEC utilizes a distributed deployment environment, configuring different servers for different regions.

[0005] Although MEC has many advantages, it also introduces some challenging problems. For example, when industrial data grows exponentially, the resources of edge servers are also limited; when multiple tasks request edge server resources at the same time, how to efficiently and reasonably allocate computing resources also needs further consideration. In response to the above problems, scholars have proposed many multi-objective optimization strategies to solve the problem of computing offloading in edge computing, such as planning reasonable migration strategies, optimizing energy consumption, or reducing service delays. However, most previous methods are offline planning of task computing migration strategies, which are not suitable for computing migration of delay-sensitive applications. Based on the above challenges, Deep Reinforcement Learning (DRL) can perform online service scheduling for complex IIoT tasks to meet its high latency constraints.

[0006] DRL uses the powerful perception capabilities of deep learning to process complex, high-dimensional environmental features, and combines the ideas of reinforcement learning to interact with the environment to complete the decision-making process. DRL can learn near-optimal decision-making strategies in real time. Although DRL's trial-and-error training is a time-consuming and resource-consuming process, once converged, the DRL agent can respond to changes in the IIoT environment within milliseconds, thereby achieving real-time decision-making. Compared with traditional heuristic algorithms, experience-driven DRL does not make any assumptions about the environmental model. It improves the strategy during learning by accumulating new experience, thereby easily adapting to environmental changes. However, how to improve the agent's learning ability and further obtain the optimal migration strategy through continuous real-time interaction with the environment remains a challenging problem.

[0007] With the continuous integration of massive numbers of industrial devices, increasing processing latency and limited local resources have become obstacles to the development of the IIoT. To fully meet the high processing demands of large-scale industrial equipment, Chen et al. (2019) studied computational offloading schemes for IIoT data processing and blockchain mining tasks, respectively, to maximize the processing efficiency of IoT devices. To minimize total cost while ensuring the stability of long-term task queues and the accuracy requirements of all task inference, Fan et al. (2023) designed a deep reinforcement learning algorithm based on a deep deterministic gradient strategy. This algorithm transforms the optimization problem into a deterministic problem for each time slot, which is then solved one by one. In the paper “Industrial Internet of Things Energy Management Architecture Based on Cloud-Mist Fusion, 2017”, Lai Chunyuan et al. proposed an IIoT energy management architecture based on cloud-mist fusion to address energy issues such as the continuous growth of energy consumption and increasingly serious environmental pollution in IIoT. This architecture realizes cloud-mist fusion and enables more reasonable and efficient utilization of cloud and fog resources.

[0008] Currently, research on optimization methods for task offloading in MEC environments primarily focuses on improving the service efficiency of edge servers. Luo et al., in "Minimizing the delay and cost of computation offloading for vehicular edge computing" (2021), studied computation offloading in vehicular edge networks and proposed an offloading framework that encompasses communication and computation with varying requirements. Their proposed framework enables efficient scheduling of edge resources and maximizes system benefits. Xu et al., in "Game theory for distributed IOV task offloading with fuzzy neural network in edge computing" (2022), studied the optimization problem of task processing delay in vehicular edge network systems. They then proposed a task offloading scheme based on fuzzy neural networks and game theory, achieving industry-leading optimization results. Xia Shichao et al., in "Distributed heterogeneous task offloading algorithm in mobile edge computing" (2019), proposed a dynamic distributed heterogeneous task offloading algorithm. Leveraging a distributed game mechanism and combining Lyapunov optimization theory, they designed a dynamic resource bidding mechanism, enabling differentiated control of different service types and on-demand allocation of computing resources.

[0009] In the paper "A DRL Agent for Jointly Optimizing Computation Offloading and Resource Allocation in MEC, 2021," Chen et al. studied the joint optimization problem of computation offloading and resource allocation in MEC. Incorporating DRL technology, they designed an algorithm based on deep deterministic policy gradients, consisting of a temporal feature extraction network composed of a one-dimensional convolutional residual block and a long short-term memory network, and a rank-based priority experience replay method. This algorithm optimizes state representation and function approximation, improving the convergence of model training and ultimately significantly reducing the task completion time and energy consumption of this joint optimization problem. In the paper "A Marine Mobile Edge Computing Offloading Method Based on Deep Reinforcement Learning, 2022," Su Xin et al. considered the complex and high-dimensional constraints imposed by the strong heterogeneity of marine information system network nodes on the optimization of marine mobile edge computing task offloading. They proposed a hierarchical classification method for marine network nodes based on multi-scale heterogeneous feature attributes and a marine mobile edge computing offloading method based on deep reinforcement learning. These methods achieve optimal offloading of computation tasks and resource optimization for marine network nodes, meeting the requirements of low-latency and high-reliability application services.

[0010] In addition to the problem of computing migration for complex IIoT tasks, for the entire system, how to obtain a reasonable and efficient computing migration strategy while meeting the energy consumption and latency requirements of the user end and further optimize the computing resources of the edge server remains a challenge. Summary of the Invention

[0011] The main purpose of this invention is to overcome the defects in the existing technology and propose an edge-enabled IIoT online computing migration method based on D3QN to optimize complex IIoT tasks, thereby reducing system energy consumption and system latency.

[0012] The present invention adopts the following technical solutions:

[0013] A D3QN-based edge-enabled IIoT online computing migration method, including:

[0014] Step 1: Abstract the current industrial environment into an edge-enabled IIoT network model, and model the tasks within the IIoT to obtain a multi-constraint task model;

[0015] Step 2: Establish a multi-constraint task computing offloading problem model including latency and energy consumption based on the power of the local device, channel transmission bandwidth, and server computing power;

[0016] Step 3: Design the D3QN network based on the multi-constrained task model and the multi-constrained task computation offloading problem model, and transform the IIoT task online computation migration problem into a Markov decision process, including state, action, and reward.

[0017] Step 4: Add noise to the D3QN network and establish the NoisyD3QN algorithm based on deep reinforcement learning according to the model to obtain the optimal task offloading decision.

[0018] Preferably, in step 1, the IIoT network model is a three-layer network model, namely the device layer, the training layer and the service layer; the device layer includes local devices; the training layer is composed of a DRL agent and a D3QN network, and the acquired device layer information is input into the network and then continuously trained and learned to obtain the optimal migration strategy, which is output to the device layer and the service layer; the service layer is a platform for devices to improve computing resources, including several edge servers and a cloud server. The tasks generated by the device layer are learned by the training layer to obtain a migration strategy, which will be executed locally and migrated to the edge server or cloud server for calculation.

[0019] Preferably, in step 1, modeling the tasks within the IIoT to obtain a multi-constraint task model specifically includes:

[0020] Step 1.1: Denote the task set as R = (r1, r2, ..., r n ), the server set is E=(e0,e1,…,e m ,e m+1 ); n represents the number of tasks; e0 represents the local device; e m+1 represents cloud servers; e1, e2,…, e m represents m edge servers;

[0021] Step 1.2: Use S=(s1,s2,…,s n ) represents the migration strategy of n tasks, when s i =e0, indicating that task r i Executed locally; when s i ∈(e1,e2,…,e m ), indicating that task r i Offload to the edge server; when s i =e m+1 When , it means task r i Migrate to cloud server for execution;

[0022] Step 1.3: Get the task set R = (r1, r2, ..., r n ), for the task set R=(r1,r2,…,r n ) to traverse and obtain the task set R=(r1,r2,…,r n ) The execution order relationship matrix X of each task in the definition of DDL time Define DDL for the system's maximum deadline constraint energy The maximum energy consumption constraint of the system is then modeled as a multi-constraint task based on the execution order of the task based on the directed acyclic graph (DAG), as shown below:

[0023]

[0024] Among them, χ i,j Is a binary number, representing the task r i and r j The execution order relationship, χ i,j =1 indicates r i is r j precursor tasks.

[0025] Preferably, the step 2 specifically includes:

[0026] Step 2.1: Make the delay of the task transmission process satisfy Among them, d i Represents task r i The amount of data that needs to be transferred; b i Represents ri The transmission channel bandwidth;

[0027] Step 2.2: Let the calculation formula of task processing delay be Among them, p i Represents task r i The total number of CPUs required; u i Indicates the execution of task r i Devices i CPU revolutions per second;

[0028] Step 2.3: Combine transmission power consumption and processing power consumption to form system energy consumption; the specific expressions of system latency and system energy consumption are as follows:

[0029]

[0030]

[0031] Among them, φ active Indicates the operating power of the local device, φ idle Represents the idle power of the local device;

[0032] Step 2.4: Taking into account the user's requirements for reducing latency and energy consumption, a multi-constrained task computation offloading problem is proposed as follows:

[0033]

[0034] Preferably, the step 3 specifically includes:

[0035] Step 3.1: Consider the interaction between tasks and servers and construct a Markov decision process; the state observed by the DRL agent at time δ is represented by a 5-tuple State = {R, X, Con, DDL time ,DDL energy}; where R represents task data information, X represents the relationship between tasks, Con represents the communication channel condition, and DDL time Indicates the maximum deadline constraint, DDL energy Represents the maximum energy consumption constraint; the action space is represented as Action={a1,a2,…,a n}, representing the execution of migrating task i to a i This action,finally outputs the action set as the system’s migration strategy;

[0036] In step 3.2, as a criterion for evaluating the efficiency of migration decisions to minimize latency and energy consumption, the reward of the DRL agent is calculated using the following formula:

[0037] Reward=-(λ1·T total +λ2·Etotal )-ρ·f(DDL time ,DDL energy ) (5)

[0038] λ1+λ2=1 (6)

[0039]

[0040] The dynamic weight parameter λ1 represents the importance of time delay in the system; the dynamic weight parameter λ2 represents the importance of energy consumption in the system; and when the time delay or energy consumption exceeds the maximum constraint, the agent will receive a penalty ρ, where ρ represents the penalty coefficient.

[0041] Preferably, the step 4 specifically includes:

[0042] Step 4.1: Initialize the evaluation network weight parameter θ, the target network weight parameter θ', the evaluation network Q(State[δ],Action[δ],θ), the target network Q(State * [δ],Action * [δ],θ') and key variables; State[δ] and Action[δ] represent the current state and action information, State * [δ] and Action * [δ] represents the next state and action information. The key variables include the batch sample number B, the experience replay pool E, the target network update frequency Step, and the learning rate W;

[0043] Step 4.2: Get the current state State[δ]; directly select an action through Action[δ] = argmaxQ(State[δ], Action[δ], θ), execute the action Action[δ] in the environment, and obtain the corresponding reward Reward[δ], update the next state State[δ+1], and store the transition (State[δ], Action[δ], Reward[δ], State[δ+1]) in the experience replay pool E;

[0044] Step 4.3: Select B samples from the experience replay pool E for network training, where the target Q value is given by the formula Calculated;

[0045] Step 4.4: Introducing the mean squared loss function To calculate the gap between the target Q value and the evaluated Q value, the back propagation strategy is used to gradually change θ according to the mean square loss function to minimize the loss;

[0046] Step 4.5: Throughout this process, add Gaussian noise to the parameter space, that is, let the network parameters θ = μ + σ⊙ξ, where θ is initially randomized and then learned from training; where μ is the mean; σ is the standard deviation; ξ is the random noise; and ⊙ is the sign of the term-by-term product.

[0047] Step 4.6: The parameters of the target network and the evaluation network are synchronized every Step; finally, the Action set is updated, which is the task migration strategy S.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] This paper adopts MEC technology to transfer the computing tasks of IIoT devices to nearby resource-rich edge servers to reduce computing delays; applies the DRL algorithm concept to solve the problem of finding the optimal task migration strategy in the MEC environment, and constructs a multi-constraint task model with delay and energy consumption limits for tasks with large data volumes and diverse data types generated by IIoT; at the same time, adopts the deep reinforcement learning algorithm D3QN, combining the advantages of Double DQN and Dueling DQN, and directly gains experience through autonomous exploration without sample information, further autonomously trains the network to obtain the optimal solution; adds a noise network to the D3QN algorithm, that is, uses the Noisy D3QN algorithm to encourage exploration through the randomness of noise and enhance robustness. This targeted exploration ensures the diversity of exploration actions and improves exploration efficiency, ultimately reducing system energy consumption and system latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the D3QN-based edge-enabled IIoT online computing migration method according to an embodiment of the present invention;

[0051] Figure 2 This is a detailed flowchart of the D3QN-based edge-enabled IIoT online computing migration method according to an embodiment of the present invention.

[0052] Figure 3 This is a diagram of the edge-enabled IIoT network model architecture of an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of multi-constraint tasks in IIoT according to an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the interaction principle between the DRL agent and the environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0056] See also Figure 1 and Figure 2 As shown, the edge-enabled IIoT online computing migration method based on D3QN of the present invention includes the following steps:

[0057] Step 1: Abstract the current industrial environment into an edge-enabled IIoT network model, and model the tasks within the IIoT to obtain a multi-constraint task model;

[0058] Step 2: Establish a multi-constraint task computing offloading problem model including latency and energy consumption based on the power of the local device, channel transmission bandwidth, and server computing power;

[0059] Step 3: Design the D3QN network based on the multi-constrained task model and the multi-constrained task computation offloading problem model, and transform the IIoT task online computation migration problem into a Markov decision process, including state, action, and reward.

[0060] Step 4: Add noise to the D3QN network and establish the NoisyD3QN algorithm based on deep reinforcement learning according to the model to obtain the optimal task offloading decision.

[0061] The step 1 is specifically implemented as follows.

[0062] See also Figure 3 As shown, in this embodiment, the industrial environment is abstracted into a three-layer network model of edge-enabled IIoT, namely the device layer, training layer, and service layer. The device layer includes on-site IIoT devices, industrial sensors, etc. (local devices). A large amount of data is generated at this layer and becomes tasks through modeling. At the same time, the local device end also has a certain computing power and can perform relatively simple tasks; the training layer is composed of DRL intelligent agents and D3QN networks. Here, the information of the device layer is first obtained, input into the network, and then continuously trained and learned to obtain the best migration strategy, which is output to the device layer and service layer; the service layer is a platform for devices to improve computing resources, including several edge servers and a cloud server. The tasks generated by the device layer are learned by the training layer to obtain a migration strategy, which will be executed locally and migrated to the edge server or cloud server for calculation.

[0063] See also Figure 4As shown, the modeling of tasks within the IIoT to obtain a multi-constraint task model specifically includes the following steps.

[0064] Step 1.1: Denote the task set as R = (r1, r2, ..., r n ), the server set is E=(e0,e1,…,e m ,e m+1 ), where e0 and e m+1 Represent the local device and cloud server respectively, e1, e2,…, e m It represents m edge servers.

[0065] Step 1.2: Use S=(s1,s2,…,s n ) represents the migration strategy of n tasks, when s i =e0, indicating that task r i Executed locally; when s i ∈(e1,e2,…,e m ), indicating that task r i Offload to the edge server; when s i =e m+1 When , it means task r i Migrate to the cloud server for execution.

[0066] Step 1.3: Get the task set R = (r1, r2, ..., r n ), for the task set R=(r1,r2,…,r n ) to traverse and obtain the task set R=(r1,r2,…,r n ) The execution order relationship matrix X of each task in the definition of DDL time ,DDL energy The maximum deadline constraint and maximum energy consumption constraint of the system are then modeled as multi-constraint tasks based on the execution order of the tasks based on the directed acyclic graph (DAG), as shown below:

[0067]

[0068] Among them, χ i,j Is a binary number, representing the task r i and r j The execution order relationship of i,j =1 means r i is r j precursor tasks.

[0069] This embodiment takes task R = (r1, r2, r3, r4, r5) as an example for explanation. The corresponding execution order relationship matrix is

[0070]

[0071] If there are latency and energy constraints on DDL time =3,DDL energy =350, the multi-constraint task is modeled as {(r1,r3),(r1,r5),(r2,r3),(r3,r4)} (3,350) .

[0072] The specific process of step 2 is as follows.

[0073] Step 2.1: Delay satisfaction of task transmission process where d i Representative task r i The amount of data that needs to be transmitted, b i It represents r i transmission channel bandwidth.

[0074] Step 2.2: The calculation formula for task processing delay is where p i Representative task r i The total number of CPUs required, u i It represents the execution of task r i Devices i The number of CPU revolutions per second.

[0075] Step 2.3: System energy consumption is composed of both transmission power consumption and processing power consumption. The specific expressions of latency and energy consumption are as follows:

[0076]

[0077]

[0078] Among them, φ active Represents the operating power of the local device, φ idle Represents the idle power of the device.

[0079] Step 2.4: In summary, the present invention takes into account the user's requirements for reducing latency and energy consumption and proposes a multi-constrained task computation offloading problem, which can be expressed as:

[0080]

[0081] See also Figure 5As shown, the deep reinforcement learning used in step 3 of the present application combines the feature extraction capability of deep learning (DL) and the decision-making capability of reinforcement learning (RL), and can directly make optimal decision output according to the input multi-dimensional data, and is widely used in dynamic decision-making, real-time prediction, simulation, game playing and other fields. It continuously interacts with the environment in real time, takes the environmental information as input to obtain the experience of failure or success to update the parameters of the decision network, and thus learns the optimal decision. The implementation process of step 3 is as follows.

[0082] Step 3.1: The present application fully considers the interaction between the task and the server, and constructs a Markov decision process. The state observed by the DRL agent at time δ is represented by a 5-tuple State={R,X,Con,DDL time ,DDL energy}, wherein R represents task data information, X represents the relationship between tasks, Con represents the communication channel condition, DDL time represents the maximum deadline constraint, and DDL energy represents the maximum energy consumption constraint. The action space can be represented as Action={a1,a2,…,a n}, which represents the action of migrating task i to a i , and finally the action set is output as the migration strategy of the system.

[0083] Step 3.2: Then, the reward of the DRL agent is calculated by the following formula, which is used as a standard to evaluate the efficiency of the migration decision, so as to achieve the purpose of minimizing the delay and energy consumption. The present application sets dynamic weight parameters λ1, λ2 to represent the importance of delay and energy consumption in the system, respectively, and when the delay or energy consumption exceeds the maximum constraint, the agent will obtain a penalty ρ, and the calculation formula is as follows:

[0084] Reward=-(λ1·T total +λ2·E total )-ρ·f(DDL time ,DDL energy ) (5)

[0085] λ1+λ2=1 (6)

[0086]

[0087] In this embodiment, D3QN is based on two neural networks. In the training neural network Q, gradient descent is used to update the parameters based on the target value for parameter update. The target value is found from the maximum value of the next step in the target neural network Q′. Unlike DQN, Double DQN does not directly find the target value for parameter update from the target neural network Q′. Instead, it finds the action corresponding to the maximum Q value in the training neural network Q and then uses this selected action to calculate the target for parameter update in the target neural network Q′. Double DQN improves on the parameter update component of DQN. Dueling DQN optimizes the algorithm by optimizing the neural network structure. Dueling DQN separates the Q value update into two parts: a state function and an action advantage function that is independent of the state. The state value and action advantage function are estimated separately, and the two are combined to form the output of the Q network. The D3QN adopted in this invention combines the advantages of Double DQN and Dueling DQN. The complete steps are as follows.

[0088] Step 4.1: Initialize the evaluation network weight parameter θ, the target network weight parameter θ', the evaluation neural network Q(State[δ],Action[δ],θ), the target neural network Q(State * [δ],Action * [δ],θ') and key variables; State[δ] and Action[δ] represent the current state and action information, State * [δ] and Action * [δ] represents the next state and action information. The key variables include the batch size B (i.e., the number of samples extracted from the experience replay pool at a time), the experience replay pool E, the target network update frequency Step, and the learning rate W.

[0089] Step 4.2: Next, obtain the current state State[δ]. Since the noise itself is random, the present invention does not use the commonly used ε-greedy strategy here. Instead, it directly selects an action through Action[δ]=argmaxQ(State[δ],Action[δ],θ), executes the action Action[δ] in the environment, and obtains the corresponding reward Reward[δ], updates the next state State[δ+1], and stores the transition (State[δ],Action[δ],Reward[δ],State[δ+1]) in the experience replay pool E.

[0090] The formula argmaxQ(State[δ],Action[δ],θ) represents taking the action that maximizes the Q value, which is a fixed formula in deep reinforcement learning.

[0091] Step 4.3: Next, B samples are selected from the aforementioned experience replay pool for network training, wherein the target Q value is calculated by the formula .

[0092] Step 4.4: Then, the mean square loss function is introduced to calculate the gap between the target Q value and the evaluated Q value, and the parameter θ is gradually changed according to the mean square loss function by using the back propagation strategy to minimize the loss.

[0093] Step 4.5: During the whole process, Gaussian noise is added to the parameter space, which is a kind of noise generated by adding a normal distribution random value with a mean of zero and a standard deviation to the input data, that is, let the network parameter θ = μ + σ ⊙ ξ, wherein μ is the mean, σ is the standard deviation, which is initially randomized, and then obtained from the training learning, ξ is the random noise, which is subject to the normal distribution N(0, 1), and ⊙ is the element-wise product symbol.

[0094] Step 4.6: The parameters of the target network and the evaluation network are synchronized once every Step steps. Finally, the Action set is updated, that is, the task migration strategy S is obtained.

[0095] The present application constructs a multi-constraint task model based on the MEC-enabled IIoT system model, and uses the deep reinforcement learning algorithm D3QN to optimize the system energy consumption and time delay. By adding Gaussian noise to the D3QN network, exploration is encouraged, robustness is enhanced, and learning efficiency is improved. Finally, while meeting the system maximum deadline and energy consumption constraints, the DRL agent continuously trains and learns to output the best task migration strategy.

[0096] The above is only a specific embodiment of the present application, but the design concept of the present application is not limited to this. Any non-essential modification of the present application using this concept shall be deemed to infringe the protection scope of the present application.

Claims

1. A D3QN-based edge-enabled IIoT online computing migration method, characterized in that: include: Step 1: Abstract the current industrial environment into an edge-enabled IIoT network model, and model the tasks within the IIoT to obtain a multi-constraint task model; Step 2: Establish a multi-constraint task computing offloading problem model including latency and energy consumption based on the power of the local device, channel transmission bandwidth, and server computing power; Step 3: Design the D3QN network based on the multi-constrained task model and the multi-constrained task computation offloading problem model, and transform the IIoT task online computation migration problem into a Markov decision process, including state, action, and reward. Step 4: Add noise to the D3QN network and establish the Noisy D3QN algorithm based on deep reinforcement learning based on the model to obtain the optimal task offloading decision; The step 3 specifically includes: Step 3.1: Consider the interaction between tasks and servers and construct a Markov decision process; the state observed by the DRL agent at time δ is represented by a 5-tuple State = {R, X, Con, DDL time ,DDL energy }; where R represents task data information, X represents the relationship between tasks, Con represents the communication channel condition, and DDL time Indicates the maximum deadline constraint, DDL energy Represents the maximum energy consumption constraint; the action space is represented as Action={a1,a2,…,a n }, representing the execution of migrating task i to a i This action,finally outputs the action set as the system’s migration strategy; In step 3.2, as a criterion for evaluating the efficiency of migration decisions to minimize latency and energy consumption, the DRL agent's reward is calculated using the following formula: Reward=-(λ1·T total +λ2·E total )-ρ·f(DDL time ,DDL energy ) (5) λ1+λ2=1 (6) Among them, the dynamic weight parameter λ1 represents the importance of delay in the system; the dynamic weight parameter λ2 represents the importance of energy consumption in the system; at the same time, when the delay or energy consumption exceeds the maximum constraint, the agent will receive a penalty ρ, where ρ represents the penalty coefficient; T total Indicates system delay; E total Indicates the system energy consumption.

2. The D3QN-based edge-enabled IIoT online computing migration method according to claim 1 is characterized in that: In step 1, the IIoT network model is a three-layer network model, namely the device layer, the training layer, and the service layer; the device layer includes local devices; The training layer consists of a DRL agent and a D3QN network. It inputs the acquired device layer information into the network and continuously trains and learns to obtain the optimal migration strategy, which is then output to the device layer and service layer. The service layer is a platform for devices to increase computing resources, including several edge servers and a cloud server. The tasks generated by the device layer are learned by the training layer to obtain migration strategies, which will be executed locally or migrated to edge servers or cloud servers for computing.

3. The D3QN-based edge-enabled IIoT online computing migration method according to claim 1 is characterized in that: In step 1, the tasks within the IIoT are modeled to obtain a multi-constraint task model, specifically including: Step 1.1: Denote the task set as R = (r1, r2, ..., r n ), the server set is E=(e0,e1,…,e m ,e m+1 ); n represents the number of tasks; e0 represents the local device; e m+1 represents cloud servers; e1, e2,…, e m represents m edge servers; Step 1.2: Use S=(s1,s2,…,s n ) represents the migration strategy of n tasks, when s i =e0, indicating that task r i Executed locally; when s i ∈(e1,e2,…,e m ), indicating that task r i Offload to the edge server; when s i =e m+1 When , it means task r i Migrate to cloud server for execution; Step 1.3: Get the task set R = (r1, r2, ..., r n ), for the task set R=(r1,r2,…,r n ) to traverse and obtain the task set R=(r1,r2,…,r n ) The execution order relationship matrix X of each task in the definition of DDL time Define DDL for the system's maximum deadline constraint energy The maximum energy consumption constraint of the system is then modeled as a multi-constraint task based on the execution order of the task based on the directed acyclic graph (DAG), as shown below: Among them, χ i,j Is a binary number, representing the task r i and r j The execution order relationship, χ i,j =1 indicates r i is r j precursor tasks.

4. The D3QN-based edge-enabled IIoT online computing migration method according to claim 3 is characterized in that: The step 2 specifically includes: Step 2.1: Make the delay of task transmission process satisfy Among them, d i Represents task r i The amount of data that needs to be transferred; b i Represents r i The transmission channel bandwidth; Step 2.2: Let the calculation formula of task processing delay be Among them, p i Represents task r i The total number of CPUs required; u i Indicates the execution of task r i Devices i CPU revolutions per second; Step 2.3: Combine transmission power consumption and processing power consumption to form system energy consumption; the specific expressions of system latency and system energy consumption are as follows: Among them, φ active Indicates the operating power of the local device, φ idle Represents the idle power of the local device; Step 2.4: Taking into account the user's requirements for reducing latency and energy consumption, a multi-constrained task computation offloading problem is proposed as follows:

5. The D3QN-based edge-enabled IIoT online computing migration method according to claim 4 is characterized in that: The step 4 specifically includes: Step 4.1: Initialize the evaluation network weight parameter θ, the target network weight parameter θ', the evaluation network Q(State[δ],Action[δ],θ), the target network Q(State * [δ],Action * [δ],θ') and key variables; State[δ] and Action[δ] represent the current state and action information, State * [δ] and Action * [δ] represents the next state and action information. The key variables include the batch sample number B, the experience replay pool E, the target network update frequency Step, and the learning rate W; Step 4.2: Get the current state State[δ]; directly select an action through Action[δ] = argmaxQ(State[δ], Action[δ], θ), execute the action Action[δ] in the environment, and obtain the corresponding reward Reward[δ], update the next state State[δ+1], and store the transition (State[δ], Action[δ], Reward[δ], State[δ+1]) in the experience replay pool E; Step 4.3: Select B samples from the experience replay pool E for network training, where the target Q value is given by the formula Calculated; Step 4.4: Introducing the mean squared loss function To calculate the gap between the target Q value and the evaluated Q value, the back propagation strategy is used to gradually change θ according to the mean square loss function to minimize the loss; Step 4.5: Throughout this process, add Gaussian noise to the parameter space, that is, let the network parameters θ = μ + σ⊙ξ, where θ is initially randomized and then learned from training; where μ is the mean; σ is the standard deviation; ξ is the random noise; and ⊙ is the sign of the term-by-term product. Step 4.6: The parameters of the target network and the evaluation network are synchronized every Step; finally, the Action set is updated, which is the task migration strategy S.

Images