Edge computing migration system for intelligent factory digital twin system

Abstract

The invention relates to an edge computing migration system for an intelligent factory digital twin system, and the system comprises a scene obtaining module which is used for obtaining the real-time data of automation equipment in a factory; wherein the real-time data comprises a running state and task information; the digital twin module is used for virtually mapping the real-time data of the automation equipment and synchronizing the state between a factory physical system and a virtual system; the model construction module is used for constructing an edge computing network time delay and energy consumption model according to the real-time data of the automation equipment; and the optimization algorithm module is used for carrying out real-time optimization on allocation of calculation tasks and resources by adopting a multi-agent deep reinforcement learning algorithm based on the edge calculation network time delay and energy consumption model. The system provided by the invention has the characteristics of low time delay and high energy efficiency, is suitable for highly dynamic industrial scenes, can optimize the scheduling of computing resources, and improves the overall operation efficiency of factories.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to an edge computing migration system for a smart factory digital twin system. Background Art

[0002] With the development of Industry 4.0, smart factories are becoming more and more popular, and digital twin technology is widely used to monitor and optimize the operation of physical systems in real time. Logistics systems in smart factories, such as AGVs, are responsible for the scheduling and execution of complex automated tasks. As the task load increases, how to optimize energy consumption and latency while ensuring efficient computing has become an important technical challenge. Existing edge computing technologies cannot fully solve the problem of efficient scheduling and resource allocation of computing tasks in dynamic environments.

[0003] At present, there are many papers and reports on edge computing that study issues through the factor of latency. On the other hand, there are fewer studies on edge computing energy consumption. If the edge network can achieve lower energy consumption at lower latency and meet the multi-factor requirements of edge network latency and energy consumption, then edge computing will be used to complete tasks that require higher immediacy and low energy consumption scenarios, or greatly improve the energy efficiency of existing edge computing networks. Summary of the invention

[0004] The present invention aims to provide an edge node computing migration architecture and configuration system based on digital twins and multi-agent reinforcement learning, using digital twin technology and multi-agent deep reinforcement learning algorithms to achieve efficient migration of computing tasks between cloud, edge, and end. The system dynamically adjusts the allocation and scheduling of computing resources through real-time monitoring of the automated logistics system in the factory environment to ensure that the system operates with low latency and high energy efficiency. The system is particularly suitable for highly dynamic industrial environments, such as the operation scenarios of AGV carts, to improve the execution efficiency of computing tasks and reduce energy consumption and computing latency.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An edge computing migration system for a digital twin system of a smart factory, comprising: a scene acquisition module, a digital twin module, a model building module, and an optimization algorithm module;

[0007] The scene acquisition module is used to acquire real-time data of automation equipment in the factory; wherein the real-time data includes: operation status and task information;

[0008] The digital twin module is used to virtually map the real-time data of the automation equipment and synchronize the status between the physical system and the virtual system of the factory;

[0009] The model building module is used to build an edge computing network delay and energy consumption model based on the real-time data of the automation equipment;

[0010] The optimization algorithm module is used to optimize the allocation of computing tasks and resources in real time based on the edge computing network latency and energy consumption model using a multi-agent deep reinforcement learning algorithm.

[0011] Optionally, the digital twin module is also used to monitor and reflect the dynamic changes of the factory in real time through digital twin technology, forming a virtual-real combination operation mechanism;

[0012] The digital twin module is updated through the model building module and passes the current factory status information to the optimization algorithm module.

[0013] Optionally, the edge computing network delay and energy consumption model includes a delay model and an energy consumption model;

[0014] The delay model includes a transmission delay sub-model, a calculation delay sub-model, and a migration delay sub-model;

[0015] The energy consumption model includes a transmission energy consumption sub-model, a calculation energy consumption sub-model, and a migration energy consumption sub-model.

[0016] Optionally, the optimization algorithm module makes intelligent decisions among different computing nodes through a multi-agent deep reinforcement learning algorithm, and performs optimal computing task allocation and resource scheduling based on the real-time requirements of the tasks;

[0017] The input of the multi-agent deep reinforcement learning algorithm includes system environment state information and agent-related parameters; the system environment state information includes task size, distance between the agent and the server, link channel gain, available computing power and available bandwidth of the server; agent-related parameters include the agent's own parameters and related constraints;

[0018] The optimization steps of the multi-agent deep reinforcement learning algorithm include: first, performing an initialization operation to set the weights of the critic network and the actor network; then entering an iterative training process, for each round, first performing environment initialization, and setting the local network environment to an initial state; secondly, in each time slot, for each agent, first performing observation and action, that is, observing the current system state, selecting and executing actions based on the state and its own action strategy, and the actions include determining the task offloading strategy, transmission power, CPU frequency allocation, and channel allocation strategy, while obtaining rewards and updating the new state to the digital twin model, and storing relevant information in the replay pool after obtaining the new state observation; then performing network update, extracting samples from the replay pool, and updating the critic network and the actor network respectively according to the samples, as well as updating the actor target network and the critic target network;

[0019] The output of the multi-agent deep reinforcement learning algorithm is the task offloading strategy of each agent, including determining whether to execute tasks locally or offload them to edge servers, as well as the corresponding transmission power, CPU frequency allocation and channel allocation strategies; the agent refers to the automation equipment in the factory that needs to offload computing tasks;

[0020] The automation equipment in the factory makes decisions on task offloading and resource scheduling based on the real-time status of the factory obtained from the digital twin module and the optimal computing task allocation and resource scheduling.

[0021] Optionally, the multi-agent deep reinforcement learning algorithm includes: a system initialization phase, a task generation phase, a state observation phase, an action selection phase, an action execution and reward acquisition phase, a system state update phase, a network update phase, and a training and convergence phase;

[0022] The system initialization phase is used to initialize the policy network and value network of each agent, set a random initial weight for each agent, and calculate the resource status of each node; wherein the agent refers to each automation device in the factory, and the resource status includes: CPU frequency, bandwidth;

[0023] The task generation stage is used for the automation device to generate a computing task in each time step and select local computing, edge server computing or cloud computing; the digital twin module will synchronize the status of each computing node in real time to provide the intelligent agent with a basis for task allocation decision-making;

[0024] The state observation phase is used for each intelligent agent to observe its own local environment state and obtain the global environment state through the digital twin module; wherein the global environment state includes: CPU frequency and bandwidth of all computing nodes;

[0025] The action selection phase is used for each agent to select an action in the action space according to the observed state; wherein the action space includes: task offloading targets, which determine whether the task is processed locally or transferred to the edge or cloud for processing; resource allocation, which selects the computing frequency of the task and the allocated bandwidth;

[0026] The execution action and reward acquisition phase is used to offload the task to the corresponding computing node according to the action selected by the agent, and update the system state; and the goal of each agent is to maximize the overall performance of the system according to the completion time and energy consumption of the task;

[0027] The reward function is as follows:

[0028] Ri = -(α×energy consumption+β×delay)

[0029] Among them, Ri represents the reward function, α and β are weight factors used to balance the impact of energy consumption and delay;

[0030] The system status update phase is used for the digital twin module to synchronize the real-time status of the system after the action is executed, and update the computing nodes and resource status for the next round of decision-making; after the system status is updated, the agent observes the environment again and uses the new status to select the next round of actions; wherein the computing nodes include: edge, automation equipment;

[0031] The network update phase is used for each agent to update the weight of the Actor network using a policy gradient method by replaying the data in the memory pool;

[0032] The training and convergence phases are used to enable the agent to interact and learn in the environment through continuous iterations, so that the reward function reaches a preset stable state and finally converges to an optimized task offloading strategy.

[0033] The beneficial effects of the present invention are:

[0034] The present invention provides an edge computing migration system for a digital twin system of a smart factory, including a scenario acquisition module for scenarios of mobile terminal computing offloading and edge node computing migration under multiple mobile terminals to multiple edge nodes, a model construction module for the delay and energy consumption model under the scenario, an optimization operation module for the total energy efficiency expression of the entire network and a combination of the configuration for the expression, and an optimization configuration module for energy efficiency optimization configuration of edge node resources according to optimal state parameters. The present invention designs a delay and energy consumption model for scenarios of mobile terminal computing offloading and edge node computing migration under multiple mobile terminals to multiple edge nodes, introduces a penalty function related to system stability, establishes an expression for the total energy efficiency of the entire network, and configures a gray wolf optimization algorithm combined with a particle swarm operator according to the optimization parameters required by the expression. The algorithm is based on the constraint relationship between various resources, and iteratively solves to obtain an edge node computing migration resource configuration scheme that optimizes energy efficiency, thereby achieving the purpose of reducing the delay and energy consumption of the entire network and improving energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 This is a schematic diagram of the structure of an edge computing migration system for a smart factory digital twin system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] This embodiment proposes an edge computing migration system for a smart factory digital twin system, including: a scene acquisition module, a digital twin module, a model construction module, and an optimization algorithm module;

[0040] The scene acquisition module is used to obtain real-time data of automation equipment in the factory; the real-time data includes: operation status and task information;

[0041] Digital twin module, which is used to virtually map the real-time data of automation equipment and synchronize the status between the factory physical system and the virtual system;

[0042] Model building module, used to build edge computing network latency and energy consumption models based on real-time data from automation equipment;

[0043] The optimization algorithm module is used to optimize the allocation of computing tasks and resources in real time based on the edge computing network latency and energy consumption model and the use of a multi-agent deep reinforcement learning algorithm.

[0044] Specifically, this embodiment can collect the operating status of the automated logistics system (such as AGV) in the factory, build a digital twin model to synchronize the status of the physical system in real time, and combine the multi-agent deep reinforcement learning algorithm to achieve efficient migration of computing tasks and resource allocation. , thereby achieving the purpose of reducing the latency and energy consumption of the entire network and improving energy efficiency.

[0045] The data flow in this embodiment is: scene acquisition module → digital twin module → model construction module → optimization algorithm module. Data is collected from the factory environment, the digital twin model is constructed and updated, and the optimization algorithm module on the AGV performs task migration and resource scheduling based on the status information provided by the digital twin, and finally feeds back to the digital twin model to ensure real-time synchronization.

[0046] The AGV makes decisions in real time through the optimization algorithm module, and the digital twin module synchronizes these changes to support the task offloading and scheduling decisions of the next AGV. Specifically,

[0047] Scene acquisition module

[0048] First, real-time data is collected from the physical environment of the smart factory, including the operating status, task load, location, and other information of AGVs and other computing nodes.

[0049] This data provides the basis for subsequent optimization and decision-making of the system, ensuring that the entire system always reflects the current status of the factory.

[0050] The scene acquisition module interacts with physical devices, collects data and passes it to the model building module for real-time construction and updating of digital twin models. The model building module is based on digital twin technology and is used to build edge computing network latency and energy consumption models based on scenarios. This ensures that AGVs and other intelligent agents can perform task scheduling and resource allocation based on accurate environmental information.

[0051] Specifically, in the model construction module of this embodiment, the digital twin model is dynamically constructed and updated according to the data provided by the scene acquisition module. The model construction module is based on digital twin technology and is used to construct an edge computing network delay and energy consumption model according to the scene.

[0052] Furthermore, the digital twin module is also used to monitor and reflect the dynamic changes of the factory in real time through digital twin technology, forming a virtual-real combination operation mechanism;

[0053] The digital twin module is updated through the model building module and passes the current factory status information to the optimization algorithm module.

[0054] Specifically, in the digital twin module of this embodiment, as a virtual mapping of the factory physical system, the state changes of computing nodes and equipment are synchronized in real time. It does not execute decisions, but provides a visualization and analysis tool to help AGV make resource scheduling decisions.

[0055] Through digital twin technology, the system can monitor and reflect the dynamic changes of the factory in real time, forming a virtual and real operating mechanism.

[0056] The digital twin model is updated through the model building module and passes the current factory status information to the optimization algorithm module on the AGV.

[0057] After the decision is executed, the state changes of the AGV will be fed back to the digital twin module to ensure the synchronization and consistency of the system.

[0058] Furthermore, the edge computing network latency and energy consumption model includes a latency model and an energy consumption model;

[0059] The delay model includes transmission delay sub-model, computation delay sub-model and migration delay sub-model;

[0060] The energy consumption model includes transmission energy consumption sub-model, calculation energy consumption sub-model and migration energy consumption sub-model.

[0061] Furthermore, the algorithm module is optimized to make intelligent decisions among different computing nodes through a multi-agent deep reinforcement learning algorithm, and to perform optimal computing task allocation and resource scheduling based on the real-time requirements of the task; computing nodes refer to nodes with computing capabilities, including AGVs and edge servers.

[0062] The input of the optimization algorithm includes system environment status information and agent-related parameters. System environment status information includes task size, distance between agent and server, link channel gain, available computing power and available bandwidth of the server, etc. Some of this information is obtained by digital twins; agent-related parameters include the agent's own parameters and related constraints.

[0063] The optimization steps are as follows: First, perform the initialization operation to set the weights of the critic network and the actor network. Then enter the iterative training process. For each round, first initialize the environment and set the local network environment to the initial state. Then in each time slot, for each agent, first observe and act, that is, observe the current system state, select and execute actions based on the state and its own action strategy. The actions include determining the task offloading strategy, transmission power, CPU frequency allocation and channel allocation strategy, and obtaining rewards and updating the new state to the digital twin. After obtaining the new state observation, store the relevant information in the replay pool; then update the network, extract samples from the replay pool, and update the critic network and actor network according to the samples, as well as the actor target network and critic target network.

[0064] The output of the algorithm is the task offloading strategy of each intelligent agent, including determining whether to execute tasks locally or offload them to edge servers, as well as the corresponding transmission power, CPU frequency allocation and channel allocation strategies. These strategies aim to minimize the total cost of the system during task execution and improve the overall performance of the system.

[0065] The automation equipment in the factory makes decisions on task offloading and resource scheduling based on the real-time status of the factory obtained from the digital twin module and the optimal computing task allocation and resource scheduling.

[0066] Furthermore, the multi-agent deep reinforcement learning algorithm includes: a system initialization phase, a task generation phase, a state observation phase, an action selection phase, an action execution and reward acquisition phase, a system state update phase, a network update phase, and a training and convergence phase;

[0067] The system initialization phase is used to initialize the policy network and value network of each agent, set a random initial weight for each agent, and calculate the resource status of each node; wherein the agent refers to each automation device in the factory, and the resource status includes: CPU frequency, bandwidth;

[0068] The task generation stage is used for the automation device to generate a computing task in each time step and select local computing, edge server computing or cloud computing; the digital twin module will synchronize the status of each computing node in real time to provide the intelligent agent with a basis for task allocation decision-making;

[0069] The state observation phase is used for each intelligent agent to observe its own local environment state and obtain the global environment state through the digital twin module; wherein the global environment state includes: CPU frequency and bandwidth of all computing nodes;

[0070] The action selection phase is used for each agent to select an action in the action space according to the observed state; wherein the action space includes: task offloading targets, which determine whether the task is processed locally or transferred to the edge or cloud for processing; resource allocation, which selects the computing frequency of the task and the allocated bandwidth;

[0071] The execution action and reward acquisition phase is used to offload the task to the corresponding computing node according to the action selected by the agent, and update the system state; and the goal of each agent is to maximize the overall performance of the system according to the completion time and energy consumption of the task;

[0072] The reward function is as follows:

[0073] Ri = -(α×energy consumption+β×delay)

[0074] Among them, Ri represents the reward function, α and β are weight factors used to balance the impact of energy consumption and delay;

[0075] The system status update phase is used for the digital twin module to synchronize the real-time status of the system after the action is executed, and update the computing nodes and resource status for the next round of decision-making; after the system status is updated, the agent observes the environment again and uses the new status to select the next round of actions; wherein the computing nodes include: edge, automation equipment;

[0076] The network update phase is used for each agent to update the weight of the Actor network using a policy gradient method by replaying the data in the memory pool;

[0077] The training and convergence phases are used to enable the agent to interact and learn in the environment through continuous iterations, so that the reward function reaches a preset stable state and finally converges to an optimized task offloading strategy.

[0078] Specifically, the optimization algorithm module of this embodiment is deployed on the AGV;

[0079] As an intelligent agent in the system, AGV is responsible for decision-making functions in resource scheduling and task migration. The optimization algorithm module uses multi-agent deep reinforcement learning to perform real-time optimization on the allocation of computing tasks and resources.

[0080] AGV makes decisions on task offloading and resource scheduling based on the real-time factory status obtained from the digital twin model to ensure the overall efficient operation of the factory.

[0081] AGV obtains the latest factory status information from the digital twin model to support task scheduling decisions.

[0082] The decision results of the optimization algorithm module affect the actions of the AGV, and these state changes will be fed back to the digital twin model to maintain system synchronization and feedback loop.

[0083] 1. Edge node computing migration for mobile terminal computing offloading

[0084] (1) Definition of Edge Node (EN)

[0085] Edge nodes are part of the edge computing network and have functions such as the transmission and calculation of computing offload tasks. Here, edge nodes include but are not limited to the following (taking the general Internet of Vehicles as an example): fixed edge nodes, such as cellular network base stations (BS), roadside communication units (RSU); mobile edge nodes, such as other vehicles joining the Internet of Vehicles.

[0086] (2) Definition of Mobile Terminal (MD) Computation Offloading and Edge Node Computation Migration

[0087] This embodiment defines as follows:

[0088] Mobile terminal computing offloading refers to the behavior of mobile terminal devices in edge computing networks making decisions to transmit tasks that have resource-sensitive factors or device factors that exceed the processing capabilities of mobile terminals through the edge network and hand them over to edge nodes for processing.

[0089] Edge node computing migration refers to the behavior of redistributing task content or results within the edge network when processing computing offload tasks of mobile terminal devices due to objective constraints such as location and energy efficiency in the edge computing network model, so that the computing offload tasks can be successfully executed and returned to the mobile terminal.

[0090] Generally, edge node computing migration is a possible step to complete the computing offload task of the mobile terminal and return the result to the mobile terminal. The occurrence conditions are as follows. For the convenience of analysis, it is assumed that the occurrence conditions are independent of each other, and the actual occurrence conditions may overlap.

[0091] In the following, for the sake of convenience, computing offloading and computing migration are assumed to refer to computing offloading to mobile terminals and computing migration to edge nodes unless otherwise specified.

[0092] (3) Calculate the conditions for migration

[0093] a. Displacement computing migration caused by the change of mobile device location: computing migration caused by the mobile terminal device crossing the edge node coverage boundary at the beginning and end of a period of time.

[0094] b. Energy-efficiency computing migration caused by edge node computing power limitation: The computing power required for the task of computing offload on the mobile terminal device causes the computing power requirement of the offload target edge node to exceed the computing power requirement, resulting in computing migration.

[0095] (4) Calculate the specific behavior of migration

[0096] The direction defined below is the vector sum of the path;

[0097] a. Displacement calculation The direction of migration is always along the direction of the mobile terminal's destination at a certain moment. This direction is uniquely determined.

[0098] b. The direction of energy efficiency calculation migration is always along the direction of the edge nodes with idle computing power. This direction is affected by various parameters of other edge nodes.

[0099] 2. Computation offloading scenario analysis

[0100] a. Computation offloading of a mobile terminal at an edge node

[0101] At this time, computation offloading occurs, and the offloading target is only the existing edge nodes.

[0102] b. Computation offloading of a mobile terminal under multiple edge nodes

[0103] At this time, computation offloading occurs, and offloading allocation is performed according to the terminal optimization decision.

[0104] c. Computation offloading of a mobile terminal through multiple edge nodes (nodes are not intersecting)

[0105] At this time, the computation offloading target is unique, and computation migration occurs in the terminal moving direction.

[0106] d. A mobile terminal passes through multiple edge nodes (node ​​crossover) for computation offloading

[0107] At this time, the computation offloading target is determined by the decision-making, and computation migration may occur.

[0108] The computation migration in ad scenarios is generally displacement computation migration.

[0109] e. Multiple mobile devices offload computations through one (or more) edge nodes.

[0110] adA collection of various situations, a general scenario in reality. Generally, the result of a large number of mobile devices gathering is the increase in the computing power requirements of edge nodes. However, due to technical, scale, funding and other reasons, the computing power of edge nodes is limited. At this time, if a large number of tasks are offloaded to the same edge node at the same time, energy-efficient computing migration will occur. Therefore, situation e is a complex combination of displacement and energy-efficient computing migration.

[0111] 3. Computation offloading and computation migration model

[0112] Assume that there are M user equipments (UE) and N edge servers (ES) in the network. In each time slot t, each user equipment u in the UE layer i Randomly generate a task, represented as (in MB). Task data can be calculated in three ways: local calculation, short-distance ES calculation, and long-distance ES calculation.

[0113] Intelligent decision-making includes two parts: one is task allocation, that is, choosing local computing, short-distance ES computing or long-distance ES computing; the other is resource scheduling (resource allocation), that is, choosing the transmission power, CPU frequency allocation and channel allocation during offloading.

[0114] Local Computing

[0115] In local computing mode, tasks are performed by the terminal device u i Local execution. i The computing resources are denoted as f i u (t). The number of CPU cycles required to perform a 1-bit computation task is denoted as ζ. In this way, the local computation execution task can be written as The calculation delay is as follows:

[0116]

[0117] 2) Close-range ES calculation

[0118] In the near-field computing mode, tasks are offloaded from end users to edge servers at the access layer via wireless communications. j The computing resources are recorded as In this way, you can write local computing execution tasks The calculation delay is as follows:

[0119]

[0120] The transmission data rate of wireless communication is determined by the available spectrum, interference, and total bandwidth. This paper adopts Orthogonal Frequency Division Multiple Access (OFDMA) as the access mode for user equipment. OFDMA can divide the system bandwidth into multiple parallel orthogonal subchannels. Due to the orthogonality between subcarriers, the subchannels will not interfere with each other. Therefore, OFDMA can achieve interference-free, parallel data transmission between multiple devices. The end user u i and nearby base stations j The transmission data rate between can be written as follows:

[0121]

[0122] Among them B ij (t). is the edge server assigned to the end user u in time slot t i bandwidth, The end user u i The transmission power in time slot t, h ij (t) is the end user u i and the current channel gain of the BS at time slot t. Calculated from the terminal user location and BS location:

[0123]

[0124] Among them, -α is the terminal user u i and servers j The path loss exponent between 2 is the noise power.

[0125] User i Wireless transmission task The time delay can be expressed as:

[0126]

[0127] Task processing delay includes task Wireless transmission delay and server j The sum of the calculation delays can be expressed as:

[0128]

[0129] 3) Long-distance ES calculation

[0130] If the task needs to be transferred from the access layer to the task transfer layer for offloading, it is defined as long-distance ES computing. The task will first be offloaded to the server s in the access layer through wireless communication.j , servers j Then transmit it to the target server s via wired communication j' Perform calculations.

[0131] User i Transfer tasks The wired transmission delay can be expressed as:

[0132]

[0133] Servers j and j' The wired transmission rate between is expressed as:

[0134]

[0135] Where: is the distance factor.

[0136] The total task processing delay includes the task Wireless transmission delay, wired transmission delay and server j' The sum of the calculation delays can be expressed as:

[0137]

[0138] C. Energy Consumption Model

[0139] The energy consumption is composed of task calculation energy consumption and task transmission energy consumption. i Indicates that during local computing, user u i Assign to task The computing resources (i.e., the number of CPU cycles per second) of a unit of computing resources are ζf i 2 , where ζ is the effective capacitance constant of the CPU chip. Therefore, the execution task The energy consumption is expressed as follows:

[0140]

[0141] in is the number of CPU cycles required to execute a unit task locally. Similarly, if the calculation is performed on the edge server, the execution task The energy consumption is expressed as follows:

[0142]

[0143] in is the number of CPU cycles required by the server to execute a unit task, f j Indicates server j Assign to task In the task transmission stage, energy consumption is mainly reflected in the uplink task transmission and the downlink calculation result transmission. Since the scale of the calculation result is much smaller than the scale of the calculation task, this paper only considers the energy consumption of the uplink task transmission. i Transmit tasks to the server via wireless communication j The energy consumption can be written as follows:

[0144]

[0145] Among them, p ij (t) is transmitted from the end user to the server s j The transmission power, r ij (t) is the end user u i and servers j The data transmission rate between.

[0146] When the task is calculated locally, there is no need to consider the transmission energy consumption, only the local computing energy consumption is considered. Then user u i The energy consumed by processing tasks is:

[0147]

[0148] When computing on a close-range server, the task is performed on servers within the wireless transmission range. j When executing, the total energy consumption is equal to the wireless transmission energy consumption and the server s j The sum of the energy consumption for computing the task is:

[0149]

[0150] In long-distance server computing, the task is outside the wireless transmission range of the server j' When executing, the energy consumption is equal to the end user transmitting the task to the nearest base station s j Wireless transmission energy consumption, base station s j To the target server j' The wired transmission energy consumption and target server s j' The sum of the energy consumption for computing the task. j To the target server j' The wired transmission energy consumption of . is very small and can be ignored, so the total energy consumption of task offloading is:

[0151]

[0152] Assume that the computing power network system has M end users and N servers, u i ∈M,s j∈ N. The basic problem in computing power network is to allocate offloading targets and computing power to each task. The basic idea is to transfer the issued tasks to the appropriate location to reduce execution delay and improve computing efficiency.

[0153] As mentioned earlier, the task It can be calculated locally, on a nearby edge server, or on a remote server. Introducing the weight factor w i Construct an objective function and adjust the weights of energy consumption and latency according to the UE's task requirements. The task execution cost consists of execution time cost and energy cost. The total energy efficiency expression of the entire network is as follows:

[0154]

[0155] Where t is the current time slot in the iterative calculation process. The task transfer problem can be expressed as follows:

[0156]

[0157] C1:

[0158] C2:

[0159] C3:

[0160] C4:

[0161] Constraint 1 represents the association strategy between the user's tasks and computing node j, x ij =0 means local calculation, x ij =j means on server s j Constraints 2 and 3 ensure that the computing resources allocated to users cannot exceed the server s j The maximum bandwidth and computing frequency of. Constraint 4 ensures that the user transmit power is less than the maximum transmit power and is non-negative. The problem is a combinatorial problem, and its optimization variables are highly complex coupled. Due to the above challenges of task transfer in computing power networks, solving the task transfer problem through existing optimization algorithms is very challenging and cannot handle dynamic system conditions well. This embodiment proposes to use multi-agent DRL to find the optimal solution to the problem.

[0162] Since task offloading is a sequential decision-making process, this embodiment models the task offloading problem as a Markov decision process (MDP). Therefore, a method based on deep reinforcement learning (DRL) can be used to solve the task offloading problem. In the system model, each vehicle (AGV) is regarded as a deep reinforcement learning (DRL) agent. Agent i takes action a in its local environment. i, and all agents take joint action a(t) = {a1(t), a2(t), ..., a m (t)}. The agent’s immediate reward is obtained based on the joint action a(t). Under action a(t), the system state s(t) evolves to the next state s(t+1) with a transition probability P(s′|s,a). Then, agent i observes the new state from its current local environment state and digital twin. The state space, action space, and reward function of the established Markov decision process (MDP) model are as follows:

[0163] State space: The state of the cloud-edge-vehicle system is determined by the task size D i (t), the distance d between the agent and the server i (t) = {d i1 (t),d i2 (t),...,d iN (t)}, the channel gain h of the link between the agent and the server i (t) = {h i1 (t),h i2 (t),...,h iN (t)}, the available computing power of all servers f(t) = {f1(t),f2(t),...,f N (t)} and the available bandwidth of all servers b(t) = {b1(t),b2(t),...,b N (t)}. The local state observed by agent i at time slot t is expressed as follows:

[0164] s i (t) = {D i (t),d i (t),h i (t),f(t),b(t)}

[0165] The system state at time slot t is expressed as follows:

[0166] s(t)={s1(t),s2(t),…,s M (t)}

[0167] Action space: At time slot t, each agent i takes actions based on its observations s i (t) and its action strategya i (t) Execute its action. This action determines the task offloading strategy x i (t), determine the target device and transmission power p for task calculation i (t), CPU frequency distribution f i (t) and channel allocation strategy b i (t). The action set can be expressed as follows:

[0168] a i (t) = {x i (t),p i (t),f i (t),b i (t)}

[0169] Reward function: Considering the collaborative learning scheme of the agent, the reward function aims to improve the overall system performance. The reward function of the agent is defined as follows:

[0170]

[0171] In the multi-agent collaboration process, the overall reward function is designed based on the global optimization goal and is defined as follows:

[0172]

[0173] where λ i Represents the weighting factor in the accumulation term.

[0174] Algorithm execution process: First, the network weights are initialized, and then the outer loop is entered for iteration. Each iteration initializes the local network environment first, and then enters the inner loop. For each time step, each agent observes the system state, selects and takes actions, obtains rewards, updates the digital twin state, and obtains new state observation values, and then stores the relevant information in the replay pool. In each iteration of the outer loop, each agent must also take samples from the replay pool to update the comment network, Actor network, and the corresponding target network. The whole process is a logic based on multi-agent reinforcement learning, and the purpose is to allow the agent to learn the optimal task offloading and resource allocation strategies in the system to improve system performance.

[0175] Example Analysis

[0176] (1) Example environment

[0177] A smart factory is equipped with M edge computing linked working AGVs and N roadside units and communication base stations (hereinafter collectively referred to as edge nodes).

[0178] (2) Parameter assumptions

[0179] For the convenience of research, it is assumed that the parameters of AGV, including CPU frequency and device transmission energy, are the same; it is assumed that the parameters of edge nodes, including CPU frequency, working voltage, edge node migration energy and bandwidth, are the same, and the edge node establishment intervals are the same. For the convenience of solving, there is no computational offloading relationship between AGVs (that is, AGVs are not used as edge nodes).

[0180] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An edge computing migration system for a smart factory digital twin system, characterized in that: include: Scene acquisition module, digital twin module, model building module, and optimization algorithm module; The scene acquisition module is used to acquire real-time data of automation equipment in the factory; wherein the real-time data includes: operation status and task information; The digital twin module is used to virtually map the real-time data of the automation equipment and synchronize the status between the physical system and the virtual system of the factory; The model building module is used to build an edge computing network delay and energy consumption model based on the real-time data of the automation equipment; The optimization algorithm module is used to optimize the allocation of computing tasks and resources in real time based on the edge computing network latency and energy consumption model using a multi-agent deep reinforcement learning algorithm.

2. The edge computing migration system for the digital twin system of a smart factory according to claim 1 is characterized in that: The digital twin module is also used to monitor and reflect the dynamic changes of the factory in real time through digital twin technology, forming a virtual-real combination operation mechanism; The digital twin module is updated through the model building module and passes the current factory status information to the optimization algorithm module.

3. The edge computing migration system for the digital twin system of a smart factory according to claim 1, characterized in that: The edge computing network delay and energy consumption model includes a delay model and an energy consumption model; The delay model includes a transmission delay sub-model, a calculation delay sub-model, and a migration delay sub-model; The energy consumption model includes a transmission energy consumption sub-model, a calculation energy consumption sub-model, and a migration energy consumption sub-model.

4. The edge computing migration system for the digital twin system of a smart factory according to claim 1, characterized in that: The optimization algorithm module uses a multi-agent deep reinforcement learning algorithm to make intelligent decisions between different computing nodes and perform optimal computing task allocation and resource scheduling based on the real-time needs of the tasks; The input of the multi-agent deep reinforcement learning algorithm includes system environment state information and agent-related parameters; the system environment state information includes task size, distance between the agent and the server, link channel gain, available computing power and available bandwidth of the server; agent-related parameters include the agent's own parameters and related constraints; The optimization steps of the multi-agent deep reinforcement learning algorithm include: first, performing an initialization operation to set the weights of the critic network and the actor network; then entering an iterative training process, for each round, first performing environment initialization, and setting the local network environment to an initial state; secondly, in each time slot, for each agent, first performing observation and action, that is, observing the current system state, selecting and executing actions based on the state and its own action strategy, and the actions include determining the task offloading strategy, transmission power, CPU frequency allocation, and channel allocation strategy, while obtaining rewards and updating the new state to the digital twin model, and storing relevant information in the replay pool after obtaining the new state observation; then performing network update, extracting samples from the replay pool, and updating the critic network and the actor network respectively according to the samples, as well as updating the actor target network and the critic target network; The output of the multi-agent deep reinforcement learning algorithm is the task offloading strategy of each agent, including determining whether to execute tasks locally or offload them to edge servers, as well as the corresponding transmission power, CPU frequency allocation and channel allocation strategies; the agent refers to the automation equipment in the factory that needs to offload computing tasks; The automation equipment in the factory makes decisions on task offloading and resource scheduling based on the real-time status of the factory obtained from the digital twin module and the optimal computing task allocation and resource scheduling.

5. The edge computing migration system for the digital twin system of a smart factory according to claim 4 is characterized in that: The multi-agent deep reinforcement learning algorithm includes: system initialization phase, task generation phase, state observation phase, action selection phase, action execution and reward acquisition phase, system state update phase, network update phase, training and convergence phase; The system initialization phase is used to initialize the policy network and value network of each intelligent agent, set a random initial weight for each intelligent agent, and calculate the resource status of each computing node; wherein the intelligent agent refers to the automation equipment in the factory that needs to offload computing tasks, and the resource status includes: CPU frequency, bandwidth; The task generation stage is used for the automation device to generate a computing task in each time step and select local computing, edge server computing or cloud computing; the digital twin module will synchronize the status of each computing node in real time to provide the intelligent agent with a basis for task allocation decision-making; The state observation phase is used for each intelligent agent to observe its own local environment state and obtain the global environment state through the digital twin module; wherein the global environment state includes: CPU frequency and bandwidth of all computing nodes; The action selection phase is used for each agent to select an action in the action space according to the observed state; wherein the action space includes: task offloading targets, which determine whether the task is processed locally or transferred to the edge or cloud for processing; resource allocation, which selects the computing frequency of the task and the allocated bandwidth; The execution action and reward acquisition phase is used to offload the task to the corresponding computing node according to the action selected by the agent, and update the system state; and the goal of each agent is to maximize the overall performance of the system according to the completion time and energy consumption of the task; The reward function is as follows: Ri = -(α×energy consumption+β×delay) Among them, Ri represents the reward function, α and β are weight factors used to balance the impact of energy consumption and delay; The system status update phase is used for the digital twin module to synchronize the real-time status of the system after the action is executed, and update the computing nodes and resource status for the next round of decision-making; after the system status is updated, the agent observes the environment again and uses the new status to select the next round of actions; wherein the computing nodes include: edge, automation equipment; The network update phase is used for each agent to update the weight of the Actor network using a policy gradient method by replaying the data in the memory pool; The training and convergence phases are used to enable the agent to interact and learn in the environment through continuous iterations, so that the reward function reaches a preset stable state and finally converges to an optimized task offloading strategy.

Images