Task unloading system of computer digital twinning auxiliary edge network

Abstract

The invention discloses a task unloading system of a computer digital twinning auxiliary edge network. The system comprises a network module, a communication module, a service cache module and a task unloading module. The network module constructs a digital twin edge computing network; the communication module divides the bandwidth of the base station and calculates the transmission rate; the service cache module determines the cache content of the base station and provides a dynamic cache mechanism; the task unloading module covers local, base station (local and cooperation) and core network processing modes, calculates related energy consumption and delay, and optimizes task completion conditions. Meanwhile, a distributed joint resource optimization algorithm based on MAPPO is established, an optimization problem is converted into a Markov decision process, system time delay and energy consumption minimization are achieved through multi-agent cooperation, and the utilization efficiency of edge network resources is improved. According to the method, system time delay and energy consumption are effectively reduced, resource utilization efficiency and task processing performance are improved, the method adapts to a dynamic network environment, and system performance and user experience are remarkably optimized.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology in computer mobile communications, and specifically to a computer digital twin-assisted edge network task offloading system. Background Art

[0002] Intelligent offloading strategies for computing tasks are crucial for effectively utilizing computing resources in edge servers. As edge terminal devices become more intelligent and advanced, optimal collaborative task offloading and service caching between user devices and edge servers can directly improve system performance.

[0003] Current research assumes that edge servers are capable of handling all types of computing tasks from mobile terminal devices, whether it is image recognition, behavior monitoring, or any other type of user request service, and can provide the best service quality. However, the actual situation is that the storage resources of edge servers are limited, so it is actually unrealistic to expect edge servers to store all services at the same time. In view of this reality, it is necessary to carefully decide the service caching strategy of each edge server, that is, which software program codes and databases should be cached to correspond to the corresponding services. This decision will directly affect whether the mobile terminal device chooses to offload tasks to the edge server, which will have a profound impact on the overall performance of the mobile edge computing (MEC) system. Therefore, in such a cache-assisted MEC system, it is particularly critical to ensure the effective coordination between service caching and task offloading.

[0004] Task offloading strategies and service caching are tightly coupled and complex problems. The computing needs of mobile terminal devices, the resource availability of edge servers, and the wireless network environment are all time-varying and dynamic, and difficult to accurately predict in practice, which brings unprecedented challenges to cloud-edge-end intelligent collaboration. Faced with the highly heterogeneous network architecture, dynamic network environment, and decentralized edge network resource allocation in the future, it is particularly important to design and implement more flexible and adaptive distributed computing offloading and resource allocation strategies. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is: how to solve the technical problems of limited edge server storage resources, coordinated optimization of task offloading and service cache, difficulty in resource scheduling in a dynamic environment, and design and implementation of distributed optimization strategies in edge computing systems.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a computer digital twin-assisted edge network task offloading system, the system comprising a network module, a communication module, a service cache module, and a task offloading module;

[0008] The network module is composed of a core network, an edge server and a user terminal device;

[0009] The communication module is used to divide the base station bandwidth into orthogonal channels, define the channel allocation decision of the user terminal equipment, calculate the uplink transmission rate, and meet the relevant constraints;

[0010] The service cache module, in which the base station can cache services and exchange cache information, also proposes a dynamic cache mechanism;

[0011] The task offloading module is used for local processing of tasks of the user terminal device or offloading them to different locations.

[0012] As a preferred solution of the task offloading system of the computer digital twin assisted edge network of the present invention, wherein: the core network is used to perform specific network functions, the edge server processes computing tasks and interacts with the physical world, and the user terminal device and the edge server are connected with the DT to form a network to achieve policy optimization and feedback;

[0013] Core network To express;

[0014] The edge server is composed of a base station BS and an MEC server. One base station BS is equipped with one MEC server. represents a base station;

[0015] Each MEC-assisted base station BS is equipped with cache and computing capabilities to Timely processing by user terminal equipment The resulting computationally intensive and latency-sensitive computing tasks

[0016] For each MEC server co-located with the base station BS, a digital twin DT of the edge server is deployed by mapping the real-time information of its physical world;

[0017] The edge server continuously monitors the state of its physical world to maintain consistency with its virtual world;

[0018] Both user terminal devices and edge servers are virtually connected to the digital twin DT through dual-to-dual communication to form a digital twin edge computing network DT-ECN;

[0019] Use the digital twin edge computing network DT-ECN to simulate and optimize offloading strategies and implement low-latency virtual twins;

[0020] Feedback the optimal strategy to the physical network and implement it.

[0021] As a preferred solution of the task offloading system of the computer digital twin assisted edge network described in the present invention, wherein: the total bandwidth of the base station in the communication module is divided into M orthogonal channels, each channel has the same number of orthogonal subcarriers, let represents a collection of channels;

[0022] For user terminal devices covered by the base station, use a u,m ∈{0,1} represents the channel allocation decision of user terminal device u, indicating whether user terminal device u is in the channel Unload its tasks;

[0023] a u,m =1 means that channel m is allocated to user terminal device u, otherwise a u,m =0.

[0024] As a preferred solution of the task offloading system of the computer digital twin assisted edge network described in the present invention, wherein: the BS in the service cache module has cache and computing capabilities, can cache services of computing-intensive and delay-sensitive applications, and allow corresponding tasks to be processed on the edge network, while reducing the frequency of requests to the core network;

[0025] Let binary variable is the service caching decision for service k in BSn at time slot t. If service k is cached at BSn, then otherwise

[0026] As a preferred solution of the task offloading system of the computer digital twin assisted edge network described in the present invention, wherein: for the task offloading module, both the user terminal device and the edge server have computing capabilities, and the computing task of the user terminal device u Can be performed locally or offloaded to local, adjacent edge servers or core networks.

[0027] As a preferred solution of the computer digital twin assisted edge network task offloading system described in the present invention, a distributed joint resource optimization algorithm based on MAPPO is further established, and the steps are as follows:

[0028] Establish MDP model;

[0029] Establish the DJCC distributed joint resource optimization algorithm framework.

[0030] As a preferred solution of the MAPPO-based distributed joint resource optimization algorithm described in the present invention, wherein: the establishment of the MDP model includes defining a state space, defining an action space, and defining a reward function;

[0031] The defined state space is used to describe the current system state, including the task queue of the user terminal device, the computing resource usage of the edge server, the cache capacity, and the spectrum resources of the communication channel;

[0032] The defined action space is used to describe the task offloading strategy of the user terminal device, including local processing, offloading to a local base station for processing, offloading to a neighboring base station for collaborative processing, or uploading to a core network for processing;

[0033] The definition of the reward function includes calculating the reward value according to the total delay and energy consumption performance of task completion, and the goal is to minimize the task processing delay and system energy consumption through optimization strategy.

[0034] As a preferred solution of the MAPPO-based distributed joint resource optimization algorithm described in the present invention, the establishment of the DJCC distributed joint resource optimization algorithm framework includes:

[0035] Design a multi-agent collaboration mechanism, treat each edge server as an independent agent, and optimize task offloading and resource allocation strategies through distributed learning;

[0036] Introducing the Actor-Critic architecture, using the Actor network to generate offloading strategies, and using the Critic network to evaluate the value of offloading strategies;

[0037] Objective function design, based on the generalized advantage estimation method, defines the objective function to balance the efficiency and stability of policy updates while minimizing system resource costs;

[0038] Centralized training and distributed execution: In the training phase, the Critic network is centrally updated based on the global state information. In the execution phase, each agent independently uses the trained Actor network to achieve efficient task decision-making.

[0039] Optimize algorithm updates and adjust the parameters of the Actor and Critic networks through the policy gradient algorithm.

[0040] Beneficial effects of the present invention: 1. A MEC system architecture assisted by digital twins is proposed, which has the characteristics of task offloading and service caching. Considering the dynamic service cache replacement mechanism, the task processing delay and energy consumption cost under three task offloading modes are calculated, and the joint optimization problem of task offloading and service caching is further expressed as a Markov decision problem, aiming to minimize the system's delay and energy consumption cost.

[0041] 2. Since the variables of service cache and task offloading constitute a high-dimensional matrix, traditional single-agent deep reinforcement learning methods face challenges. In order to obtain the optimal strategy and adapt to the dynamic network environment, this patent proposes a hierarchical task offloading and resource allocation joint optimization scheme based on multi-agent proximal policy optimization algorithm (DJCCMAPPO). This scheme uses the improved MAPPO algorithm to train the model and obtain the optimal task offloading and resource allocation strategy to minimize system latency and energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0043] Figure 1 A system operation flow chart of a task offloading system of a computer digital twin-assisted edge network of the present invention;

[0044] Figure 2 A collaborative service cache and task offloading framework diagram in a DT-assisted MEC system in a computer digital twin-assisted edge network task offloading system of the present invention;

[0045] Figure 3 It is a diagram of three task offloading modes in a task offloading system of a computer digital twin assisted edge network of the present invention;

[0046] Figure 4 This is a framework diagram of the DJCC algorithm in a task offloading system of a computer digital twin-assisted edge network of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0048] Example, see Figures 1 to 4 , is an embodiment of the present invention, and provides a computer digital twin-assisted edge network task offloading system, such as Figure 1 As shown, the system includes:

[0049] Network module, communication module, service cache module, task offloading module.

[0050] The network module consists of three layers: core network, edge server, and user terminal equipment.

[0051] The network module consists of three layers: core network, edge server and user terminal device. The core network performs specific network functions, the edge server processes computing tasks and interacts with the physical world, and the user terminal device and edge server are connected to the DT to form a network to achieve policy optimization and feedback;

[0052] The communication module divides the base station bandwidth into orthogonal channels, defines the channel allocation decision of the user terminal equipment, calculates the uplink transmission rate, and satisfies the relevant constraints;

[0053] The service cache module, in which the base station can cache services, but due to space limitations, the cache content needs to be determined, and cache information can be exchanged at the same time, and a dynamic cache mechanism is also proposed;

[0054] The task offloading module, in which the tasks of the user terminal device can be processed locally or offloaded to different locations, has multiple processing methods and corresponding energy consumption and delay expressions, and is intended to optimize task completion under resource constraints.

[0055] Furthermore, the network module consists of three layers: core network, edge server, and user terminal equipment. The edge server is composed of a base station (BS) and a MEC server. One base station is equipped with one MEC server. Represents a base station. Each MEC-assisted BS is equipped with a cache and computing capabilities to Timely processing by user terminal equipment The resulting computationally intensive and latency-sensitive computing tasks For each MEC server co-located with the BS, DT is deployed by mapping real-time information of its physical world, such as the number, location, and requested task status of user terminal devices associated with the base station, as well as the communication, storage, and computing resources of each edge server. The edge server continuously monitors the status of its physical world to maintain consistency with its virtual world. Both user terminal devices and edge servers are virtually connected to DT through two-to-two communications to form a digital twin edge computing network (DT-ECN). Taking advantage of DT-ECN, it is easy to simulate and optimize offloading strategies and achieve low-latency virtual twins. Finally, the optimal strategy is fed back to the physical network and implemented.

[0056] In each time slot t, the computational tasks generated by the user terminal device u served by the BS are Represented as an ordered vector c u where d represents the total computational effort of the task; u Indicates the data size of the task, including programs / databases / libraries; Represents the service type, which is a collection of service catalogs; Indicates the processing delay tolerance of the task, that is, the total processing delay of the task is not allowed to exceed In addition, the definition As including The set of task offloading decisions for all user terminal devices in is further described as Where (1) α u = 1 or 0 determines whether the task is processed locally; (2) β u = 1 or 0 determines whether to offload the tasks of the user terminal equipment to the local base station for processing; (3) γ u = 1 or 0 determines whether to offload the tasks of the user terminal equipment to the adjacent base station for processing; (4)δ u =1 or 0 determines whether to offload the tasks of the user terminal equipment to the core network for processing;

[0057]

[0058] At time slot t, each user terminal device requests the service to be completed and expects to achieve the minimum latency and energy consumption cost. The user terminal device can perform the task with its own computing resources, such as device 1 in edge clusters 1 and 2 to perform application services locally. In addition, the device can offload the task to the local edge server, such as devices 2 and 3. Once the software related to the task is cached in the local edge server, the task can be processed in time when the server receives the data, such as device 2 in edge cluster 1; otherwise, the additional cost of downloading the software is required, such as device 3 in edge cluster 1 or devices 2 and 4 in edge cluster 2, where the edge server needs to wait until the software has been obtained from the neighboring edge base station or core network. Since the computing power of devices and edge servers is different and limited, reasonable decisions should be made to make full use of all computing resources on the edge side to minimize system costs.

[0059] Furthermore, the total bandwidth of the base station in the communication module is divided into M orthogonal channels, each channel has the same number of orthogonal subcarriers, let represents a set of channels. Due to the orthogonality of channels in the OFDMA system, there is no interference between the channels of the BS. However, co-channel interference still exists, that is, the channel is interfered by the user terminal devices covered by other base stations on the same channel. For the user terminal devices covered by the base station, use a u,m ∈{0,1} represents the channel allocation decision of user terminal device u, indicating whether user terminal device u is in the channel Unload its tasks on the u,m =1 means that channel m is allocated to user terminal device u, otherwise a u,m = 0. Based on the Shannon capacity formula, the uplink transmission rate of the user terminal device u on the channel m in its local base station n can be expressed as:

[0060]

[0061] Where B is the bandwidth of the wireless transmission channel, p u,m and h u They represent the transmission power of user terminal device u on channel m and the channel gain between user terminal device u and BS respectively. The channel gain is given by Where L u represents the distance between the user terminal device and the BS, μ = 4 is the path loss; σ 2 is the noise power, is the receiving interference from other user terminal devices u to BSn on channel m, further expressed as where h u' is the interference channel gain from user terminal device u to BS n on channel m.

[0062] Since each BS adopts the OFDMA scheme, each user terminal device can be assigned at most one channel, so the following constraints must be met:

[0063]

[0064] In addition, each channel can only be assigned to one user terminal device at most, that is:

[0065]

[0066] Based on the above assumptions, the uplink transmission rate of user terminal device u in its local BS n can be expressed as:

[0067]

[0068] Furthermore, the BS in the service cache module has caching and computing capabilities, which can cache services of computationally intensive and delay-sensitive applications and allow the corresponding tasks to be processed on the edge network, while reducing the frequency of requests to the core network. However, not all services can be cached on the BS with limited storage capacity at the same time. Therefore, the BS needs to wisely decide which types of services should be cached. Let the binary variable is the service caching decision for service k in BSn at time slot t, where if service k is cached at BSn, then otherwise In addition, it is assumed that the services cached during time slot t can be used to perform the corresponding tasks in the next time slot t+1. Considering that the total cached services cannot exceed the storage capacity of the BS, the following capacity constraints apply:

[0069]

[0070] Among them, Z n is the storage capacity of BSn, Z k Is a cache service Required storage space. In addition, this patent assumes that all BSs in the network can exchange service cache information at the beginning of each time slot.

[0071] Due to the limitation of BS storage space, BS caches limited computing services. However, user terminal devices generate different tasks, so the requested computing services vary over time.

[0072] In order to ensure effective task offloading, this patent proposes a dynamic service caching mechanism to determine which services need to be cached in the BS. This mechanism enables the BS to flexibly request service types from neighboring edge servers and core networks according to different tasks. set up Represents a computing task The requested service k, when When computing tasks Request service k; if Then service k is not available in the edge server. On this basis, we define The popularity of the computing service is based on the number of requests. The dynamic service cache replacement mechanism is shown in Table 1.

[0073] Table 1 Dynamic service cache replacement mechanism process

[0074]

[0075]

[0076] Furthermore, for the task offloading module, both the user terminal device and the edge server have computing capabilities, so the computing task of the user terminal device u Can be performed locally or offloaded to local, adjacent edge servers or core networks.

[0077] S1-1, local processing, α u =1,β u =0,γ u =0,δ u = 0. When the service component corresponding to the task is cached in the terminal device, only local computing delay is generated, and no transmission delay is generated. The computing delay expression is:

[0078]

[0079] Among them, f u is the local CPU cycle frequency.

[0080] The corresponding calculation energy consumption expression is:

[0081]

[0082] in, It is an energy coefficient related to the chip architecture of the user terminal device.

[0083] S1-2, base station processing

[0084] In this section, BSs cooperate to provide computing services to user terminals. Specifically, when offloading to the base station to process tasks, there are two offloading strategies for each user terminal:

[0085] S1-2-1, base station local processing: the task can be offloaded to the MEC server co-located with the local BS for execution, i.e.

[0086] α u =0,β u =1,γ u =0,δ u =0;

[0087] Once the user terminal device decides to offload its task to the local BS, the task will first be uploaded to the BS and then executed at the edge server co-located with the BS. After the local base station completes the task, it returns the calculation result to the corresponding user terminal device. In this patent, the focus is mainly on the uplink transmission delay and the calculation delay of the task.

[0088] The calculation delay expression of the base station local processing is:

[0089]

[0090] The transmission delay expression of the base station local processing is:

[0091]

[0092] The calculation energy consumption expression of the base station local processing is:

[0093] Where ψ is the energy coefficient related to the chip architecture of the local BS.

[0094] The transmission energy consumption expression of the base station local processing is:

[0095]

[0096] S1-2-2, base station cooperative processing: For a local BS that does not cache the requested service type or does not provide sufficient computing power, the local BS further migrates the task to one of the other BSs that has cached the service for execution, i.e., α u =0,β u =0,γ u =1,δ u =0.

[0097] Migrating tasks to another BS will cause additional transmission delay. In order to focus on the study and simplify the system model, is the average transmission rate between BSn and BSn'. Therefore, the calculation delay expression for sending the calculation task of user terminal device u to BSn' is:

[0098]

[0099] The transmission delay expression of base station cooperative processing is:

[0100]

[0101] The computational energy consumption expression of base station cooperative processing is:

[0102]

[0103] The transmission energy consumption expression of base station cooperative processing is:

[0104]

[0105] S1-3, core network processing, α u =0,β u =0,γ u =0,δ u =1.

[0106] When the service cache component does not exist in the terminal device and the MEC server, the task needs to be uploaded to the core network for processing. Considering the rich computing power of the core network, the core network processing can ignore the task calculation delay and energy consumption, and only consider the transmission delay and energy consumption. The transmission delay of the task of the core network processing the user terminal device u can be expressed as:

[0107]

[0108] in Indicates the data transmission rate of the wired connection from the BS to the cloud server.

[0109] The transmission energy consumption expression of the core network processing is:

[0110]

[0111] The total task completion delay consists of transmission delay and computation delay. The expression is:

[0112]

[0113] The total task completion energy consumption consists of transmission energy consumption and computation energy consumption. The expression is:

[0114]

[0115] In order to jointly minimize task completion delay and energy consumption under resource constraints and make optimal content caching and task offloading decisions, the constraint objective is determined as:

[0116]

[0117]

[0118] Constraint C1 indicates that the task offloading decision is a binary variable, constraint C2 indicates that a task can only choose one offloading mode, constraint C3 indicates that the cellular channel allocation decision is a binary variable, constraint C4 is expressed as an indicator of whether the service component is cached, constraint C5 is a binary indicator of whether the cached service component is requested, constraint C6 is a binary decision variable of whether the cached service component is replaced, constraints C7 and C8 indicate the uniqueness of the channel allocation decision, constraint C9 ensures that the total data size of the cached service cannot exceed the total storage of the edge server, and constraint C10 indicates the guarantee that the processing delay of task u must not exceed the task processing delay tolerance.

[0119] like Figure 2 As shown, the present invention establishes a distributed joint resource optimization algorithm based on MAPPO, and the specific steps are as follows:

[0120] S2-1, MDP model.

[0121] S2-2, DJCC distributed joint resource optimization algorithm framework.

[0122] S2-3, description of DJCC distributed joint resource optimization algorithm.

[0123] Furthermore, in the MDP model of S2-1, in the edge network environment, due to the characteristics of dense heterogeneity of network structure, highly dynamic network environment and decentralized resources, the centralized management method cannot cope well with the highly dynamic MEC environment, and a more dynamic and flexible distributed task offloading and service caching strategy needs to be designed. Based on the main idea of ​​deep reinforcement learning, the above optimization problem is transformed into a Markov decision process (MDP) to represent the hierarchical task offloading and multi-dimensional resource allocation decision process.

[0124] The present invention regards each edge server as an independent agent to maximize the reward function. Each edge server provides services to terminal devices with insufficient computing power and no cached corresponding service components in a distributed manner. This collaborative multi-agent optimization problem is posed by Description, where Represents the global state space of the environment, including edge server computing resources, cache capacity, etc. The joint action space of all edge servers is given by Indicates that a e represents the action of edge server e. In each time slot t, each edge server adopts strategy π t,e (a t,e |s t,e ) takes action, and the actions of all edge servers cause To the next state The global state transition follows the state transition probability Rewards It means that the common optimization goal of all edge servers is to maximize the cumulative discounted total benefit where χ is the discount factor.

[0125] To adapt the optimization problem to the MADRL framework, the basic elements of the MDP are as follows:

[0126] (1) Statespace: In this section, the state space is used to reflect the environment, which includes the task queue and the amount of idle computing resources, cache capacity, and spectrum resources of each edge server at the current time slot t. The state space is defined as:

[0127]

[0128] Therefore, a set of system state spaces can be defined as:

[0129] (2) Actionspace: At time slot t, action a(t) contains the offloading strategy selected by the agent for the terminal device. The agent offloads tasks to neighboring terminal devices, edge servers, neighboring edge servers, and remote cloud servers by obtaining resources from the terminal device, edge server, and remote cloud server and deciding whether to execute locally. For the agent, at the beginning of each time slot, tasks u are collected within its coverage area. For the terminal device d, it can only choose one way to process its task i in each time slot. The decision variable about the offloading mode is α u ,β u ,γ u ,δ u ∈{0,1}, so define α u +β u +γ u +δ u ∈{0, 1}. For the agent, the action space is defined as:

[0130]

[0131] Then the system action space can be defined as:

[0132] (3) Reward function: The reward function measures the effect of an action taken by the agent in a given state. During the training process, if the agent takes an action in the t-1 time slot, the corresponding reward will be returned to the agent in the t time slot. Based on the reward obtained, the agent will update its strategy to obtain the best result. While completing the computational task, the agent should also minimize the system cost. Therefore, the system reward function can be defined as:

[0133]

[0134] Furthermore, in step S2-2, in order to solve the above multi-agent MDP model, a DJCC distributed joint resource optimization algorithm is proposed. Multiple edge servers work together, and each edge server is composed of an actor module, a critic module, and an experience storage.

[0135] Based on the MAPPO collaborative task offloading optimization algorithm, this paper designs a new objective function to achieve small batch update. Let π θ represents the actor network used to approximate the policy, and V ωrepresents the critic network used to approximate the value, where θ and ω represent the parameters of the actor network and the critic network in the MEC server, respectively. The pruning agent objective function of the collaborative task offloading optimization algorithm based on MAPPO is expressed as follows:

[0136]

[0137] Among them, ρ t (θ)=π θ (a t,e |s t,e ) / π θdd (a t,e |s t,e ) represents the ratio of the new strategy to the old strategy in the edge server, and ε represents the clip score. The main function of the clip function is to avoid excessive modification of the target value. represents the generalized advantage estimator (GAE), which is expressed as follows:

[0138]

[0139] Among them, H e represents the current edge server and neighboring agents, π t,e' represents the neighboring agent strategy at time t.

[0140] The critic network in each edge server is parameterized by ω and learned using gradient descent with a loss function as follows:

[0141] L VF (ω)=E t [(V ω (s t )-y(t)) 2 ](4.27)

[0142] The target value expression is y(t)=r t +μV ω (s t +1).

[0143] In the single-agent PPO algorithm, the original work proposed to use the entropy reward X[π θ ] to enhance the overall objective function to promote exploration. The above loss function (4.29) is combined with the entropy reward X[π θ ] Adding them together, the total objective function is:

[0144] L total =E t [L CLIP (θ)-c1L VF (ω)+c2X[πθ ](s t )] (4.28)

[0145] where c1 and c2 are coefficients, and X[π θ ](s t ) indicates that the input state s t Provides the entropy of the policy.

[0146] Multi-agent reinforcement learning can provide a distributed perspective for edge servers. The DJCC distributed joint resource optimization framework includes E agents and environments, where each MEC server acts as an agent to execute the PPO algorithm. The operation process of each agent is divided into two stages: centralized training and distributed execution. In the centralized training stage, the optimal strategy is explored offline to ensure that all agents can work together to achieve the global optimum. In the execution stage, the agent uses the learned strategy for forward propagation without random exploration, thereby improving the efficiency and accuracy of decision-making.

[0147] In the centralized training phase, the critic network acts as a central coordinator and calculates the centralized action-value function Q(s,a1,a2,...,a e |ω), including the actions and observations of all agents. The centralized Q function evaluates the Actor actions from a global perspective and guides them to choose better actions. Then, the critic network updates the parameters θ by minimizing the loss function c :

[0148] loss(θ c )=E[Q(s,a1,a2,...,a e |ω)-y MAPPO ] 2

[0149]

[0150] where s=(s1,s2,...,s e ),ω is the parameter of the critic network. s' represents the updated state of the target network, and ω' is the updated parameter of the evaluation network. On the other hand, the actor network updates the network parameters θ and outputs actions based on the centralized Q function calculated by the critic network and its own observations. Specifically, the actor network adjusts the network parameters θ according to the following formula. The global state achieves faster and simpler value learning through fully observable MDP by assuming a centralized value function:

[0151]

[0152] In the distributed execution phase, the critic network is not included, and only the trained actor network works online. The decentralized edge server makes decisions based only on its own local observations. Only one forward propagation process is performed during the entire execution process, which greatly reduces the time consumption and computing resource consumption compared to the training phase.

[0153] Specifically, network training consists of two parts:

[0154] S3-1, experience collection stage.

[0155] S3-2, strategy update phase.

[0156] In the S3-1 experience collection phase, each edge server uses a shared strategy to select an action and collect experience until the maximum time step T is reached. Then, the reward function and advantage function are calculated based on them respectively. In addition, the strategy and value function of each MEC server in each time period are updated in an experience replay manner based on the DJCC distributed joint resource optimization framework model. The PPO algorithm used by each MEC server is trained to estimate the parameter gradient direction towards the maximum total weighted reward. For each training step, first calculate

[0157] In the S3-2 strategy update phase, the actor network π t,e Using the loss function L CLIP (θ)Optimize Epoch, and the critic Critic network V ω Using the loss function L VF (ω) Optimize the same mini-batch of data sampled from the storage buffer D times. To ensure that the training process is stable, the data samples are randomly renumbered and cleaned to break the correlation between them (line 14). ω The network structure and actor network π t,e The network structure is the same as that of , except that the last layer has only one output value. The optimization tool used is the Adam optimizer. The DJCC distributed joint resource optimization algorithm process is shown in Table 2.

[0158] Table 2 DJCC distributed joint resource optimization algorithm process

[0159]

[0160]

[0161] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A computer digital twin-assisted edge network task offloading system, characterized in that: The system includes a network module, a communication module, a service cache module, and a task offloading module; The network module is composed of a core network, an edge server and a user terminal device; The communication module is used to divide the base station bandwidth into orthogonal channels, define the channel allocation decision of the user terminal equipment, calculate the uplink transmission rate, and meet the relevant constraints; The service cache module, in which the base station can cache services and exchange cache information, also proposes a dynamic cache mechanism; The task offloading module is used for local processing of tasks of the user terminal device or offloading them to different locations.

2. The computer digital twin-assisted edge network task offloading system according to claim 1, characterized in that: The core network is used to perform specific network functions, the edge server processes computing tasks and interacts with the physical world, and the user terminal device and the edge server are connected with the DT to form a network to achieve policy optimization and feedback; Core network To express; The edge server is composed of a base station BS and an MEC server. One base station BS is equipped with one MEC server. represents a base station; Each MEC-assisted base station BS is equipped with cache and computing capabilities to Timely processing by user terminal equipment The resulting computationally intensive and latency-sensitive computing tasks For each MEC server co-located with the base station BS, a digital twin DT of the edge server is deployed by mapping the real-time information of its physical world; The edge server continuously monitors the state of its physical world to maintain consistency with its virtual world; Both user terminal devices and edge servers are virtually connected to the digital twin DT through dual-to-dual communication to form a digital twin edge computing network DT-ECN; Use the digital twin edge computing network DT-ECN to simulate and optimize offloading strategies and implement low-latency virtual twins; Feedback the optimal strategy to the physical network and implement it.

3. The computer digital twin-assisted edge network task offloading system according to claim 2, characterized in that: The total bandwidth of the base station in the communication module is divided into M orthogonal channels, each channel has the same number of orthogonal subcarriers. represents a collection of channels; For user terminal devices covered by the base station, use a u,m ∈{0,1} represents the channel allocation decision of user terminal device u, indicating whether user terminal device u is in the channel Unload its tasks; a u,m =1 means that channel m is allocated to user terminal device u, otherwise a u,m =0.

4. The computer digital twin-assisted edge network task offloading system according to claim 3, characterized in that: The BS in the service cache module has caching and computing capabilities, can cache services of computing-intensive and delay-sensitive applications, and allow corresponding tasks to be processed on the edge network, while reducing the frequency of requests to the core network; Let binary variable is the service caching decision for service k in BSn at time slot t. If service k is cached at BSn, then otherwise 5. The computer digital twin-assisted edge network task offloading system according to claim 4, characterized in that: For the task offloading module, both the user terminal device and the edge server have computing capabilities, and the computing task of the user terminal device u Can be performed locally or offloaded to local, adjacent edge servers or core networks.

6. The computer digital twin-assisted edge network task offloading system according to claim 5, characterized in that: Further establish a distributed joint resource optimization algorithm based on MAPPO, the steps are: Establish MDP model; Establish the DJCC distributed joint resource optimization algorithm framework.

7. The MAPPO-based distributed joint resource optimization algorithm according to claim 6, characterized in that: The establishing of the MDP model includes defining a state space, defining an action space, and defining a reward function; The defined state space is used to describe the current system state, including the task queue of the user terminal device, the computing resource usage of the edge server, the cache capacity, and the spectrum resources of the communication channel; The defined action space is used to describe the task offloading strategy of the user terminal device, including local processing, offloading to a local base station for processing, offloading to a neighboring base station for collaborative processing, or uploading to a core network for processing; The definition of the reward function includes calculating the reward value according to the total delay and energy consumption performance of task completion, and the goal is to minimize the task processing delay and system energy consumption through optimization strategy.

8. The MAPPO-based distributed joint resource optimization algorithm according to claim 6, characterized in that: The establishment of the DJCC distributed joint resource optimization algorithm framework includes: Design a multi-agent collaboration mechanism, treat each edge server as an independent agent, and optimize task offloading and resource allocation strategies through distributed learning; Introducing the Actor-Critic architecture, using the Actor network to generate offloading strategies, and using the Critic network to evaluate the value of offloading strategies; Objective function design, based on the generalized advantage estimation method, defines the objective function to balance the efficiency and stability of policy updates while minimizing system resource costs; Centralized training and distributed execution: In the training phase, the Critic network is centrally updated based on the global state information. In the execution phase, each agent independently uses the trained Actor network to achieve efficient task decision-making. Optimize algorithm updates and adjust the parameters of the Actor and Critic networks through the policy gradient algorithm.

Images