Service caching and computing unloading optimization method under digital twin cloud edge network

Abstract

The invention provides a service caching and computing unloading optimization method under a digital twin cloud edge network, which comprises a virtual digital twin cloud edge network model DTCEN and a service caching and computing unloading joint optimization method JCO-CR based on convex optimization enabling deep reinforcement learning. The DTCEN comprises a system controller SyC, a cloud computing center CC, I base stations BSs and J user equipment UDs; the BS set is marked as # imgabs0 #, and the UD set is marked as # imgabs1 #; each BS is provided with an MEC server used for providing storage and computing resources, and the storage and computing resources of the # imgabs2 # are respectively marked as # imgabs3 # and # imgabs4 #; the UDs are randomly distributed in the communication coverage range of each BS, and the nearest BS is used as a local BS; the number of the UDs in the communication coverage range of the # imgabs5 # is recorded as # imgabs6 #, wherein the # imgabs7 #; the number of service types is K, and the service types are recorded as # imgabs8 #; the SyC is responsible for collecting system state information and transmitting the system state information to the DRL agent, and the DRL agent makes a service caching and calculation unloading decision; according to the method, the service delay can be effectively reduced, and more excellent performance can be shown in different scenes.

Description

Technical Field

[0001] The present invention relates to the field of network topology technology, and in particular to a method for optimizing service caching and computing offloading in a digital twin cloud-edge network. Background Art

[0002] With the development of 5G communication technology and the popularity of mobile devices, network traffic has shown explosive growth. At the same time, various emerging intelligent applications continue to emerge, such as virtual reality / augmented reality (VR / AR), object detection and autonomous driving, which are usually computationally intensive and delay-sensitive. Cloud computing responds to users' growing service needs by deploying server clusters in remote data centers, but there is a contradiction between long-distance communication and the real-time requirements of intelligent applications. To alleviate this problem, the European Telecommunications Standards Institute (ETSI) introduced a new distributed computing paradigm: Mobile Edge Computing (MEC). By deploying computing resources at the edge of the network, users can offload the tasks of their intelligent applications to MEC servers to avoid excessive communication delays caused by interacting with the remote cloud. In addition, the processing of tasks depends on the corresponding input data and specific service programs, so the effective use of caching technology to cache services in advance can further improve the quality of service (Quality-of-Service, QoS).

[0003] Edge caching technology can be mainly divided into two categories: content caching and service caching. Specifically, content caching caches the output results of tasks in MEC servers or user devices to save task processing delays; service caching caches the service programs required for task processing in MEC servers or user devices to save task transmission delays. Usually, there are large differences in the output results of different tasks, resulting in a low content cache hit rate. In contrast, pre-cached service programs can be reused, such as face recognition. However, the storage capacity of MEC servers is limited, and how to reasonably cache service programs to better meet user needs is a challenging problem. At the same time, independent caching of MEC servers may lead to low utilization of edge resources. How to coordinate different MEC servers for collaborative caching is an issue worthy of in-depth study. In addition, MEC server computing resources are limited, and how to efficiently allocate computing resources is an important issue that needs to be solved.

[0004] Existing solutions mainly use control theory or optimization theory to solve the service caching and computing offloading problems. Control theory introduces mathematical models or dynamic equations, which usually requires fixing user requirements or staticizing the MEC system, ignoring the complexity of the system and the dynamic nature of the task, which may cause its strategy to be unsuitable for real-world scenarios. Optimization theory uses multiple rounds of iterations to approximate feasible solutions, which may lead to high time complexity. It is usually difficult to make optimal decisions efficiently when facing large-scale service caching and computing offloading problems. As an emerging branch in the field of machine learning, Deep Reinforcement Learning (DRL) has been gradually applied to service caching and computing offloading problems in edge scenarios. The value-based DRL method can show good performance when facing a small action space, but when facing a large-scale action space, the performance is usually degraded due to overestimation of the Q value. The policy-based DRL algorithm is sensitive to the selection of hyperparameters and may not be effectively applicable to ubiquitous networks. At the same time, this type of DRL algorithm usually uses experience replay for sample reuse, which may lead to excessive memory consumption and the inability of old samples to adapt to dynamic and changing environments. In addition, some studies have attempted to optimize caching, offloading, and resource allocation issues simultaneously, but the huge action space can lead to unstable strategies and make them prone to falling into local optimality, thus affecting system performance and resource utilization efficiency.

[0005] It is worth noting that when DRL agents interact with the MEC environment for decision-making, they need to maintain communication with the MEC server and user devices to obtain real-time system status information and collect training samples, which may lead to excessive communication costs and low training efficiency in real environments. The emergence of Digital Twin (DT) provides a new perspective for solving this problem. It realizes the connection between the physical world and the digital space and can simulate, predict and analyze the real MEC environment. The combination of DT and MEC forms a new computing paradigm, which enables DRL agents to train models in the DT environment, explore different action strategies to learn the best decision model, and then feed the model back to the real MEC environment. If DT is not introduced, the training of DRL agents will face the timeliness problem of data collection in the physical MEC network. For this reason, edge nodes can usually only communicate continuously with user devices within their coverage to meet the training needs of DRL agents. In contrast, the combination of DT and MEC not only improves the training efficiency of DRL agents, but also reduces model training costs and system overhead.

[0006] Based on the above analysis, in order to solve the above important challenges, the present invention constructs a new digital twin cloud-edge network (DT Cloud-Edge Networks, DTCEN) model, and proposes a novel joint optimization of service caching and computation offloading with Convex-optimization-enabled deep reinforcement learning (Joint optimization of service Caching and computation Offloading with Convex-optimization-enabled deep Reinforcement learning, JCO-CR) method. A large number of simulation experiments have verified the feasibility and effectiveness of the proposed JCO-CR method. The experimental results show that compared with the other 6 benchmark methods, the JCO-CR method can effectively reduce service latency and show superior performance in different scenarios. Summary of the invention

[0007] The present invention proposes a service caching and computing offloading optimization method under the digital twin cloud-edge network, which can effectively reduce service latency and demonstrate superior performance in different scenarios.

[0008] The present invention adopts the following technical solutions.

[0009] A service caching and computation offloading optimization method under a digital twin cloud-edge network, the method comprising a virtual digital twin cloud-edge network (DT Cloud-Edge Networks, DTCEN) model DTCEN, and also comprising a service caching and computation offloading joint optimization (Joint optimization of service Caching and computation offloading with Convex-optimization-enabled deep Reinforcement learning, JCO-CR) method JCO-CR, DTCEN comprising a system controller SyC (System Controller, SyC), a cloud computing center CC (Cloud Center, CC), I base stations BSs (Base Stations, BSs) and J user devices UDs (User Devices, UDs); the BS set is denoted as B = {b1, b2, ..., b i ,…,b I}, the UD set is recorded as U = {u1,u2,…u j ,…u J Each BS is equipped with a MEC server for providing storage and computing resources, where b iThe storage and computing resources are denoted as C i and F i UDs are randomly distributed within the communication coverage of each BS and take the BS with the nearest neighbor as the local BS; b i The number of UDs within the communication coverage is denoted as n i ,in There are K types of services, denoted as S = {s1, s2, ..., s k ,…,s K SyC is responsible for collecting system status information and transmitting it to the DRL agent, which then makes decisions on service caching and computation offloading.

[0010] In the physical cloud-edge network corresponding to DTCEN, CC and BSs collect system operation status information and build a DTCEN model to achieve the mapping between the physical cloud-edge network and the virtual twin network, and continuously monitor the physical network status to ensure the consistency of the twin model with it;

[0011] The DRL agent interacts with the DTCEN environment to continuously optimize the service cache and computing offloading strategy, and then feeds the learned model back to the physical cloud-edge network to improve training efficiency;

[0012] When the dynamics of the capture system needs to be optimized, the system operation time is discretized into T equal-length time slots, denoted as T = {1, 2, ..., t, ..., T}; u j Generate a task request in each time slot, defined as a triplet task j = <d j ,cy j ,cs j >,j∈J, where d j Indicates the amount of data, cy j Indicates the calculation task j The number of CPU cycles required per bit of data, cs j Indicates the type of service that the task depends on;

[0013] When task j Unloaded to b i When b i Cached tasks j Required service type cs j And can allocate sufficient computing resources to it, task j It can be executed smoothly; otherwise, task j Need to be offloaded to other BSs or CCs for execution.

[0014] If the UDs and CC communicate over long distances, and transferring all user tasks to the CC for execution may cause severe congestion in the core network and excessive service delays, the service cache and computation offloading strategies are optimized to minimize long-term service delays. The specific methods are as follows:

[0015] Method 3.1, Cache model optimization, is used to enable the DRL agent to make appropriate service cache decisions in each time slot: At time slot t, the system service cache state is defined as Where m i,k ∈{0,1} represents b i The service cache status in; when m i,k =1, b i Cached service types k and occupies c k storage capacity; otherwise, service type s k Not cached; b i The total capacity of the cache service type cannot exceed the size of its storage space. The cache capacity constraint is expressed as:

[0016]

[0017] Among them, c k Indicates the service type k The required cache capacity, C i Indicates b i The storage space limit;

[0018] In this method, the service type cached in time slot t can be used to process the task request in time slot t+1, that is, the previous time slot b i The service cache decision is used as the service cache state of the next time slot; in DTCEN, each BS broadcasts cache information at the beginning of each time slot to achieve collaborative caching between BSs, thereby effectively reducing redundant service caches in the system;

[0019] Method 3.2, communication model optimization, in this method, user equipment u j task j There are two ways to handle this: Method 1: j Offload to local BS for processing;

[0020] Wireless communication is used between user devices and local BS. When multiple user devices send task requests at the same time, BS will use frequency division multiple access (FDMA) technology to allocate independent channels for them. According to Shannon's formula, u i With b i The transmission rate between is defined as:

[0021]

[0022] Among them, w j,i Indicates b i Assign to u i The channel bandwidth, p j Indicates u j Transmission power, g j,i Indicates u j With b i The channel gain between 2 represents Gaussian noise power; I j',i =∑p j' g j',i ,j'∈J / {j} represents the interference caused by other devices to the communication; therefore, the task upload delay is defined as:

[0023] Method 2: If the required service type is not cached on the local BS, the task j Forwarded to cooperative BSs or CC for processing; when a task is generated, its input data needs to be uploaded from the user equipment to the local BS or forwarded;

[0024] If task j Required service type cs j If the data is not cached in the local BS, it needs to be forwarded to the cooperative BSs or CC. Fiber communication is used between BSs and between BSs and CC. The number of communication hops between the local BS and the target BS is recorded as h. i,z , the length of a single-hop path is denoted as d i,z ; Let z represent the set of BSs and CCs, i.e. z∈I∪{c}; In addition, the communication transmission rate between BSs is denoted by R I,I , b i The communication transmission rate between CC is denoted as R I,c ; Therefore, the task forwarding delay is defined as:

[0025]

[0026] Where v represents the propagation velocity of light in a vacuum, ε represents the refractive index of the optical fiber, and the task communication delay is defined as:

[0027]

[0028] Method 3.3, calculation model optimization, specifically: do not consider some task offloading, that is, task j Can only be completely offloaded to one node for processing; task j The unloading decision is denoted as O j ∈I∪{0}, where Oj =i means the task is offloaded to b i , O j = 0 means the task is offloaded to the remote CC; the user devices executing the task on node z are denoted as set X z ; To ensure the smooth execution of the task request, node z needs to cache the service type required by the task and allocate an appropriate amount of computing resources to the task; Since the computing resources equipped by a single BS are limited, there are the following resource constraints, which are expressed as:

[0029]

[0030] Among them, node z is assigned to u j The computing resource ratio is denoted as f z,j , the computational delay of a task is defined as:

[0031]

[0032] Among them, when z∈I, Indicates that the task is in b i The calculation performed on z = c, Indicates the computational delay of the task executed on the CC; the proportion of computing resources allocated to each task is subject to the following constraints:

[0033] f min_z ≤f z,j ≤1,z∈I∪{c}Formula 8;

[0034] Considering the communication and computation delays, task j The service delay is defined as:

[0035]

[0036] The method explores the optimal service cache, computation offloading and resource allocation strategy to minimize long-term service delay, which is formalized as an optimization problem and expressed in the following formula:

[0037]

[0038] Among them, X z represents the set of user devices that offload tasks to node z in time slot t; C1 represents the service cache decision; C2 represents the task offloading decision; C3 represents that the sum of the capacity of the service types cached by the BS cannot exceed its storage capacity upper limit; C4 represents the node z assigned to task j The computing resource ratio constraint; C5 means that the sum of the computing resource ratios allocated to tasks by node z cannot exceed 1;

[0039] Considering the discrete decision space of service caching and computation offloading, the continuous decision space of resource allocation, and the nonlinear constraints in the optimization problem, P1 is the original optimization problem and is regarded as a mixed-integer nonlinear programming (MINLP) problem to be solved.

[0040] When solving problem P1, the system first performs model training and decision-making in DTCEN through the improved DRL method; then, in each time slot t, the service cache and computing offloading decisions are fed back to the physical cloud-edge network, and the task is offloaded to the target node; after determining the set of tasks that each node needs to process, the resource allocation plan for each node is further determined. Service cache, computing offloading and resource allocation belong to two different dimensions of problems in P1, which are solved in a serial manner. By decoupling P1, it is split into sub-problems P2 and P3 for separate solutions.

[0041] The sub-problem P2 is specifically: to minimize the long-term service delay, jointly optimize the service cache and computation offloading strategy, which is expressed as:

[0042]

[0043] The sub-problem P3 is as follows: minimize the computational delay of node z by optimizing resource allocation; after determining the service cache and computation offloading strategies, the resource allocation process of each node is relatively independent; only the resource allocation strategy of a single node needs to be optimized independently, that is, in represents the task communication delay and is independent of the resource allocation process; therefore, P3 is simplified to:

[0044]

[0045] The optimization method targets subproblem P2 and uses an improved DRL method to perform model training in DTCEN to learn the optimal service cache and computation offloading strategy, and then deploys the decision feedback to the physical cloud-edge network. That is, the exploration and learning efficiency of the DRL agent is improved by introducing DTCEN. Specifically:

[0046] The DRL agent obtains the DTCEN environment state s at time slot t t , output action a t And get the reward signal r t , then go to the next state s t+1 , the process is expressed as MDP; specifically, the state space, action space and reward function are defined as follows:

[0047] State space: state s tContains task information from the user's device. t , BSs’ cache status information M t And the number of tasks N received by each BS in the current time slot t ; Therefore, the system state at time slot t is expressed as:

[0048] s t ={Task t ,M t ,N t}, Formula XIII;

[0049] Among them, task information Task t ={D t ,CY t ,CS t ,L t}, cache status information Number of tasks

[0050] Action space: action a t Including service cache decision Cache t and task offloading decision Off t ; Therefore, the action at time slot t is expressed as:

[0051] a t = {Cache t ,Off t},Formula 14;

[0052] Among them, service cache decision Task offloading decision Reward function: Assuming the optimization goal is to minimize the long-term service delay, the reward function at time slot t is defined as:

[0053]

[0054] Wherein, J represents a set of user devices.

[0055] In the optimization method, the Actor and Critic networks are trained by the Proximal Policy Optimization (PPO) method to make optimal service caching and computation offloading decisions in the dynamically changing DTCEN; in PPO, the Actor network outputs actions under a given state and updates the policy network parameters by maximizing the expected total return; the Critic network evaluates the value of the policy and updates the evaluation network parameters by comparing the difference between the actual and predicted returns.

[0056] The proximal strategy optimization method PRO is specifically as follows:

[0057] The method is expressed in pseudocode as follows:

[0058]

[0059]

[0060] In this pseudocode: First, initialize the Actor network π θ and Critic Network V ω and its network parameters θ and ω, the experience pool M, the number of training rounds E, the number of time slots T, and the number of training rounds epochs for each update (lines 1-2); randomly initialize the service type cached by each BS until its storage capacity upper bound is reached (line 3); in each training round, the user equipment uses the nearest neighbor BS as the local BS and accesses the DTCEN through it (lines 5-6); then, the system state s at time slot t is t Enter the Actor Network π θ , generating service cache and computation offloading decision a t , and then based on the cache decision Cache t Update system status, according to uninstall decision Off t The task is unloaded to the target node (lines 8-9), where the exploration noise ε is referenced to avoid falling into the local optimum; then, each node uses convex optimization theory to derive the optimal resource allocation to minimize the task calculation delay (lines 10-11); then, the system feeds back the immediate reward r t And accumulate to the current round reward ep_re, and transfer to the next state s t+1 (Lines 13-14), and then the training samples (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool M (line 15); then, N samples are sampled from M to calculate the temporal difference (TD) error TD (lines 18-19), and the TD error is defined as:

[0061] at l =r l +γV(s l+1 ,ω)-V(s l ,ω),l∈N Formula 16;

[0062] Among them, V(s l+1 ,ω) and V(s l ,ω) respectively represent the Critic network parameter ω in state s l+1 and l The state value function under ;

[0063] Then, the advantage function of N samples is calculated based on the TD error and target value Used for subsequent network parameter updates (lines 20-22); the Actor network uses the old and new strategies to calculate the ratio p of the probability of a specific action under a given strategy t , and according to the objective function J CLIP (θ) calculates the loss function and then passes Update strategy (lines 24-25); objective function J CLIP (θ) is expressed as:

[0064]

[0065] Critic network uses objective function J Value (w) Calculate the loss function and use gradient descent to update the network parameters ω (line 26), where J Value (w) is expressed as:

[0066]

[0067] in, is the actual state value of sample i, V(s i ,ω) is the state s i The state value estimated by the network parameter ω is obtained; by minimizing the mean square error loss, the Critic network parameter ω gradually improves its estimation accuracy of the state value; after each training episode, the samples in M ​​are cleared (line 28).

[0068] After the DRL agent makes service caching and computation offloading decisions, all tasks in DTCEN have been assigned to each node to wait for execution; each node reasonably allocates its computing resources to ensure the smooth execution of tasks; that is, for the optimization problem P3, first prove that P3 is a convex optimization problem; then, based on the convex optimization resource allocation method, the optimal resource allocation for task requests in DTCEN is achieved under given caching and offloading decisions;

[0069] The proof that P3 is a convex optimization problem is:

[0070] It is necessary to prove that the objective function and all constraints of P3 are convex functions. From Formula 12, we can see that constraints C4 and C5 are affine functions and therefore convex functions. Next, we need to prove the convexity of the objective function. To simplify the description, constraint C4 is rewritten, and P3 can be transformed into:

[0071]

[0072] Among them, C6 and C7 represent the upper and lower bounds of the proportion of computing resources that can be allocated to a single task respectively; in order to prove that the objective function is a convex function, the Hessian matrix is ​​introduced, which describes the local curvature of the function; the calculation method of each element of the Hessian matrix of the objective function is as follows:

[0073] Among them, C6 and C7 represent the upper and lower bounds of the proportion of computing resources that can be allocated to a single task respectively; in order to prove that the objective function is a convex function, the Hessian matrix is ​​introduced, which describes the local curvature of the function; the calculation method of each element of the Hessian matrix of the objective function is as follows:

[0074]

[0075] Among them, d j 、cy j 、F z They represent the data volume and computing density of the task and the computing resources of node z respectively, all of which are non-negative numbers. In addition, since 0<f min_z ≤f z,j ≤1, so Both have f z,j >0; According to the above analysis, it is proved that P3 is a convex optimization problem; then, the Lagrangian function is defined as follows:

[0076]

[0077] Among them, φ, γ j and j∈X z are the numbers of Lagrange multipliers corresponding to C4, C5 and C6 respectively; for a convex optimization problem, if a point x in the definition domain satisfies the KKT condition, that is, satisfies the gradient, complementary relaxation and constraint conditions, then the point is the global optimal solution; for P3, its KKT condition is expressed as follows:

[0078]

[0079] Where t∈T, j∈X z , z∈I∪{c}; from (b) and (c) γ j (f z,j -1)=0, we can get formula (e) and (f) γ j =0; Then, by deriving from formula (a), we can obtain formula (g): Substituting it into (b), we can get (h) Finally, substituting formula (h) into formula (g) yields the optimal computing resource allocation solution set for node z:

[0080]

[0081] In each time slot t, node z can derive the optimal computing resource allocation for the task according to the above process, and then guide the DRL agent to further optimize the service cache and computing offloading strategy.

[0082] The use process of the method comprises the following steps:

[0083] Step S1, all user equipments in DTCEN access DTCEN through local BS;

[0084] Step S2: The DRL agent receives the task information from the user based on the relevant information of the user's device, including the task information sent by the user. t , BSs’ cache status information M t And the number of tasks N received by each BS in the current time slot t , to generate service caching and computation offloading decisions;

[0085] Step S3: SyC updates the cache of the service entity set on each BS according to the generated service cache decision; SyC allocates each task to the corresponding node for execution according to the generated computation offloading decision;

[0086] Step S4: Each node allocates computing resources to all tasks to be executed in the node according to a resource allocation method based on convex optimization to support their parallel operation;

[0087] Step S5: The system records the status of each time slot, the actions generated, the rewards obtained, and the new status entered. The DRL agent generates new service caching and computation offloading decisions based on the above information.

[0088] The present invention proposes JOC-CR, which is a novel service caching and computation offloading framework based on convex optimization-enabled deep reinforcement learning. First, the emerging digital twin (DT) technology is introduced to realize the mapping of physical network to virtual world, and a new digital twin cloud-edge network (DT Cloud-Edge Networks, DTCEN) model is constructed. Through the cloud-edge collaboration in DTCEN, the training efficiency is effectively improved and the resource efficiency and load balancing are improved. In addition, the original optimization problem is decoupled into two serial sub-problems from different dimensions and solved separately to reduce the complexity of the problem and improve the solution efficiency. For the service caching and computation offloading sub-problems, an improved DRL method is designed, the state space, action space and reward function of the service caching and computation offloading problem are defined, and the sub-optimization problem is formalized as a Markov decision process (MDP). The proposed method alleviates the instability of training by controlling the policy update amplitude. Different from the experience replay mechanism, the agent is directly updated on the current sampled data, making the training more lightweight and efficient, and then making fast and effective service caching and computation offloading decisions. For the resource allocation sub-problem, a resource allocation method based on convex optimization is designed. First, a constraint model is established and convex optimization theory is used to prove that it is a convex programming problem. Then, the optimal solution is theoretically derived based on the KKT (Karush-Kuhn-Tucker) condition. Finally, it is combined with the improved DRL to form the JCO-CR method, which can effectively deal with the impact of dynamic task requests on resource allocation. A large number of simulation experiments show that the performance of the JCO-CR method is better than other benchmark methods in scenarios with different BS storage capacity and different BS computing resources. Compared with the advanced DDPG and TD3 methods, the JCO-CR method also shows a more stable and superior performance level, further proving its superiority. The above results verify the feasibility and effectiveness of the proposed JCO-CR method in reducing the long-term service delay of the system.

[0089] Compared with other benchmark methods, the JCO-CR method proposed in this paper reasonably decouples the original optimization problem, and then designs a joint optimization strategy of service caching, computation offloading and resource allocation based on improved DRL and convex optimization theory, which effectively improves the performance of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0091] Attached Figure 1 It is a schematic diagram of the digital twin cloud-edge network;

[0092] Attached Figure 2It is a schematic diagram of the service cache and computation offloading method based on the improved DRL;

[0093] Attached Figure 3 It is a schematic diagram comparing the convergence of different methods;

[0094] Attached Figure 4 is a schematic diagram of the effect of BS cache capacity on the performance of different methods;

[0095] Attached Figure 5 It is a schematic diagram of the impact of BS computing resources on the performance of different methods;

[0096] Attached Figure 6 It is a schematic diagram comparing JCO-CR with other methods on different types of delays. DETAILED DESCRIPTION

[0097] like Figure 1 As shown, a service caching and computation offloading optimization method under a digital twin cloud-edge network includes a virtual digital twin cloud-edge network (DT Cloud-Edge Networks, DTCEN) model DTCEN, and also includes a service caching and computation offloading joint optimization (Joint optimization of service Caching and computation Offloading with Convex-optimization-enabled deep Reinforcement learning, JCO-CR) method JCO-CR. The DTCEN includes a system controller SyC (System Controller, SyC), a cloud computing center CC (Cloud Center, CC), I base stations BSs (Base Stations, BSs) and J user devices UDs (User Devices, UDs); the BS set is denoted as B = {b1, b2, …, b i ,…,b I}, the UD set is recorded as U = {u1,u2,…u j ,…u J Each BS is equipped with a MEC server for providing storage and computing resources, where b i The storage and computing resources are denoted as C i and F i UDs are randomly distributed within the communication coverage of each BS and take the BS with the nearest neighbor as the local BS; b i The number of UDs within the communication coverage is denoted as n i ,in There are K types of services, denoted as S = {s1, s2, ..., s k ,…,s K}; SyC is responsible for collecting system status information and transmitting it to the DRL agent, which makes service caching and computation offloading decisions.

[0098] In the physical cloud-edge network corresponding to DTCEN, CC and BSs collect system operation status information and build a DTCEN model to achieve the mapping between the physical cloud-edge network and the virtual twin network, and continuously monitor the physical network status to ensure the consistency of the twin model with it;

[0099] The DRL agent interacts with the DTCEN environment to continuously optimize the service cache and computing offloading strategy, and then feeds the learned model back to the physical cloud-edge network to improve training efficiency;

[0100] When the dynamics of the capture system needs to be optimized, the system operation time is discretized into T equal-length time slots, denoted as T = {1, 2, ..., t, ..., T}; u j Generate a task request in each time slot, defined as a triplet task j = <d j ,cy j ,cs j >,j∈J, where d j Indicates the amount of data, cy j Indicates the calculation task j The number of CPU cycles required per bit of data, cs j Indicates the type of service that the task depends on;

[0101] When task j Unloaded to b i When b i Cached tasks j Required service type cs j And can allocate sufficient computing resources to it, task j It can be executed smoothly; otherwise, task j Need to be offloaded to other BSs or CCs for execution.

[0102] If the UDs and CC communicate over long distances, and transferring all user tasks to the CC for execution may cause severe congestion in the core network and excessive service delays, the service cache and computation offloading strategies are optimized to minimize long-term service delays. The specific methods are as follows:

[0103] Method 3.1, Cache model optimization, is used to enable the DRL agent to make appropriate service cache decisions in each time slot: At time slot t, the system service cache state is defined as Where m i,k ∈{0,1} represents b i The service cache status in; when m i,k =1, b i Cached service types k and occupies c k storage capacity; otherwise, service type s k Not cached; b i The total capacity of the cache service type cannot exceed the size of its storage space. The cache capacity constraint is expressed as:

[0104]

[0105] Among them, c k Indicates the service type k The required cache capacity, C i Indicates b i The storage space limit;

[0106] In this method, the service type cached in time slot t can be used to process the task request in time slot t+1, that is, the previous time slot b i The service cache decision is used as the service cache state of the next time slot; in DTCEN, each BS broadcasts cache information at the beginning of each time slot to achieve collaborative caching between BSs, thereby effectively reducing redundant service caches in the system;

[0107] Method 3.2, communication model optimization, in this method, user equipment u j task j There are two ways to handle this: Method 1: j Offload to local BS for processing;

[0108] Wireless communication is used between user devices and local BS. When multiple user devices send task requests at the same time, BS will use frequency division multiple access (FDMA) technology to allocate independent channels for them. According to Shannon's formula, u i With b i The transmission rate between is defined as:

[0109]

[0110] Among them, w j,i Indicates b i Assign to u i The channel bandwidth, p j Indicates u j Transmission power, g j,i Indicates uj With b i The channel gain between 2 represents Gaussian noise power; I j',i =∑p j' g j',i ,j'∈J / {j} represents the interference caused by other devices to the communication; therefore, the task upload delay is defined as:

[0111] Method 2: If the required service type is not cached on the local BS, the task j Forwarded to cooperative BSs or CC for processing; when a task is generated, its input data needs to be uploaded from the user equipment to the local BS or forwarded;

[0112] If task j Required service type cs j If the data is not cached in the local BS, it needs to be forwarded to the cooperative BSs or CC. Fiber communication is used between BSs and between BSs and CC. The number of communication hops between the local BS and the target BS is recorded as h. i,z , the length of a single-hop path is denoted as d i,z ; Let z represent the set of BSs and CCs, i.e. z∈I∪{c}; In addition, the communication transmission rate between BSs is denoted by R I,I , b i The communication transmission rate between CC is denoted as R I,c ; Therefore, the task forwarding delay is defined as:

[0113]

[0114] Where v represents the propagation velocity of light in a vacuum, ε represents the refractive index of the optical fiber, and the task communication delay is defined as:

[0115]

[0116] Method 3.3, calculation model optimization, specifically: do not consider some task offloading, that is, task j Can only be completely offloaded to one node for processing; task j The unloading decision is denoted as O j ∈I∪{0}, where O j =i means the task is offloaded to b i , O j = 0 means the task is offloaded to the remote CC; the user devices executing the task on node z are denoted as set X z; To ensure the smooth execution of the task request, node z needs to cache the service type required by the task and allocate an appropriate amount of computing resources to the task; Since the computing resources equipped by a single BS are limited, there are the following resource constraints, which are expressed as:

[0117]

[0118] Among them, node z is assigned to u j The computing resource ratio is denoted as f z,j , the computational delay of a task is defined as:

[0119]

[0120] Among them, when z∈I, Indicates that the task is in b i The calculation performed on z = c, Indicates the computational delay of the task executed on the CC; the proportion of computing resources allocated to each task is subject to the following constraints:

[0121] f min_z ≤f z,j ≤1,z∈I∪{c}Formula 8;

[0122] Considering the communication and computation delays, task j The service delay is defined as:

[0123]

[0124] The method explores the optimal service cache, computation offloading and resource allocation strategy to minimize long-term service delay, which is formalized as an optimization problem and expressed in the following formula:

[0125]

[0126] Among them, X z represents the set of user devices that offload tasks to node z in time slot t; C1 represents the service cache decision; C2 represents the task offloading decision; C3 represents that the sum of the capacity of the service types cached by the BS cannot exceed its storage capacity upper limit; C4 represents the node z assigned to task j The computing resource ratio constraint; C5 means that the sum of the computing resource ratios allocated to tasks by node z cannot exceed 1;

[0127] Considering the discrete decision space of service caching and computation offloading, the continuous decision space of resource allocation, and the nonlinear constraints in the optimization problem, P1 is the original optimization problem and is regarded as a mixed-integer nonlinear programming (MINLP) problem to be solved.

[0128] When solving problem P1, the system first performs model training and decision-making in DTCEN through the improved DRL method; then, in each time slot t, the service cache and computing offloading decisions are fed back to the physical cloud-edge network, and the task is offloaded to the target node; after determining the set of tasks that each node needs to process, the resource allocation plan for each node is further determined. Service cache, computing offloading and resource allocation belong to two different dimensions of problems in P1, which are solved in a serial manner. By decoupling P1, it is split into sub-problems P2 and P3 for separate solutions.

[0129] The sub-problem P2 is specifically: to minimize the long-term service delay, jointly optimize the service cache and computation offloading strategy, which is expressed as:

[0130]

[0131] The sub-problem P3 is as follows: minimize the computational delay of node z by optimizing resource allocation; after determining the service cache and computation offloading strategies, the resource allocation process of each node is relatively independent; only the resource allocation strategy of a single node needs to be optimized independently, that is, in represents the task communication delay and is independent of the resource allocation process; therefore, P3 is simplified to:

[0132]

[0133] The optimization method targets subproblem P2 and uses an improved DRL method to perform model training in DTCEN to learn the optimal service cache and computation offloading strategy, and then deploys the decision feedback to the physical cloud-edge network. That is, the exploration and learning efficiency of the DRL agent is improved by introducing DTCEN. Specifically:

[0134] like Figure 2 As shown, the DRL agent obtains the DTCEN environment state s at time slot t t , output action a t And get the reward signal r t , then go to the next state s t+1 , the process is expressed as MDP; specifically, the state space, action space and reward function are defined as follows:

[0135] State space: state st Contains task information from the user's device. t , BSs’ cache status information M t And the number of tasks N received by each BS in the current time slot t ; Therefore, the system state at time slot t is expressed as:

[0136] s t ={Task t ,M t ,N t}, Formula XIII;

[0137] Among them, task information Task t ={D t ,CY t ,CS t ,L t}, cache status information Number of tasks

[0138] Action space: action a t Including service cache decision Cache t and task offloading decision Off t ; Therefore, the action at time slot t is expressed as:

[0139] a t = {Cache t ,Off t},Formula 14;

[0140] Among them, service cache decision Task offloading decision Reward function: Assuming the optimization goal is to minimize the long-term service delay, the reward function at time slot t is defined as:

[0141]

[0142] Wherein, J represents a set of user devices.

[0143] In the optimization method, the Actor and Critic networks are trained by the Proximal Policy Optimization (PPO) method to make optimal service caching and computation offloading decisions in the dynamically changing DTCEN; in PPO, the Actor network outputs actions under a given state and updates the policy network parameters by maximizing the expected total return; the Critic network evaluates the value of the policy and updates the evaluation network parameters by comparing the difference between the actual and predicted returns.

[0144] The proximal strategy optimization method PRO is specifically as follows:

[0145] The method is expressed in pseudocode as follows:

[0146]

[0147]

[0148] In this pseudocode: First, initialize the Actor network π θ and Critic Network V ω and its network parameters θ and ω, the experience pool M, the number of training rounds E, the number of time slots T, and the number of training rounds epochs for each update (lines 1-2); randomly initialize the service type cached by each BS until its storage capacity upper bound is reached (line 3); in each training round, the user equipment uses the nearest neighbor BS as the local BS and accesses the DTCEN through it (lines 5-6); then, the system state s at time slot t is t Enter the Actor Network π θ , generating service cache and computation offloading decision a t , and then based on the cache decision Cache t Update system status, according to uninstall decision Off t The task is unloaded to the target node (lines 8-9), where the exploration noise ε is referenced to avoid falling into the local optimum; then, each node uses convex optimization theory to derive the optimal resource allocation to minimize the task calculation delay (lines 10-11); then, the system feeds back the immediate reward r t And accumulate to the current round reward ep_re, and transfer to the next state s t+1 (Lines 13-14), and then the training samples (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool M (line 15); then, N samples are sampled from M to calculate the temporal difference (TD) error TD (lines 18-19), and the TD error is defined as:

[0149] at l =r l +γV(s l+1 ,ω)-V(s l ,ω),l∈N Formula 16;

[0150] Among them, V(s l+1 ,ω) and V(s l ,ω) respectively represent the Critic network parameter ω in state s l+1 and l The state value function under ;

[0151] Then, the advantage function of N samples is calculated based on the TD error and target value Used for subsequent network parameter updates (lines 20-22); the Actor network uses the old and new strategies to calculate the ratio p of the probability of a specific action under a given strategy t , and according to the objective function J CLIP (θ) calculates the loss function and then passes Update strategy (lines 24-25); objective function J CLIP (θ) is expressed as:

[0152]

[0153] Critic network uses objective function J Value (w) Calculate the loss function and use gradient descent to update the network parameters ω (line 26), where J Value (w) is expressed as:

[0154]

[0155] in, is the actual state value of sample i, V(s i ,ω) is the state s i The state value estimated by the network parameter ω is obtained; by minimizing the mean square error loss, the Critic network parameter ω gradually improves its estimation accuracy of the state value; after each training episode, the samples in M ​​are cleared (line 28).

[0156] After the DRL agent makes service caching and computation offloading decisions, all tasks in DTCEN have been assigned to each node to wait for execution; each node reasonably allocates its computing resources to ensure the smooth execution of tasks; that is, for the optimization problem P3, first prove that P3 is a convex optimization problem; then, based on the convex optimization resource allocation method, the optimal resource allocation for task requests in DTCEN is achieved under given caching and offloading decisions;

[0157] The proof that P3 is a convex optimization problem is:

[0158] It is necessary to prove that the objective function and all constraints of P3 are convex functions. From Formula 12, we can see that constraints C4 and C5 are affine functions and therefore convex functions. Next, we need to prove the convexity of the objective function. To simplify the description, constraint C4 is rewritten, and P3 can be transformed into:

[0159]

[0160] Among them, C6 and C7 represent the upper and lower bounds of the proportion of computing resources that can be allocated to a single task respectively; in order to prove that the objective function is a convex function, the Hessian matrix is ​​introduced, which describes the local curvature of the function; the calculation method of each element of the Hessian matrix of the objective function is as follows:

[0161] Among them, C6 and C7 represent the upper and lower bounds of the proportion of computing resources that can be allocated to a single task respectively; in order to prove that the objective function is a convex function, the Hessian matrix is ​​introduced, which describes the local curvature of the function; the calculation method of each element of the Hessian matrix of the objective function is as follows:

[0162]

[0163] Among them, d j 、cy j 、F z They represent the data volume and computing density of the task and the computing resources of node z respectively, all of which are non-negative numbers. In addition, since 0<f min_z ≤f z,j ≤1, so Both have f z,j >0; According to the above analysis, it is proved that P3 is a convex optimization problem; then, the Lagrangian function is defined as follows:

[0164]

[0165] Among them, φ, γ j and j∈X z are the numbers of Lagrange multipliers corresponding to C4, C5 and C6 respectively; for a convex optimization problem, if a point x in the definition domain satisfies the KKT condition, that is, satisfies the gradient, complementary relaxation and constraint conditions, then the point is the global optimal solution; for P3, its KKT condition is expressed as follows:

[0166]

[0167] Where t∈T, j∈X z , z∈I∪{c}; from (b) and (c) γ j (f z,j -1)=0, we can get formula (e) and (f) γ j =0; Then, by deriving from formula (a), we can obtain formula (g): Substituting it into (b), we can get (h) Finally, substituting formula (h) into formula (g) yields the optimal computing resource allocation solution set for node z:

[0168]

[0169] In each time slot t, node z can derive the optimal computing resource allocation for the task according to the above process, and then guide the DRL agent to further optimize the service cache and computing offloading strategy.

[0170] The use process of the method comprises the following steps:

[0171] Step S1, all user equipments in DTCEN access DTCEN through local BS;

[0172] Step S2: The DRL agent receives the task information from the user based on the relevant information of the user's device, including the task information sent by the user. t , BSs’ cache status information M t And the number of tasks N received by each BS in the current time slot t , to generate service caching and computation offloading decisions;

[0173] Step S3: SyC updates the cache of the service entity set on each BS according to the generated service cache decision; SyC allocates each task to the corresponding node for execution according to the generated computation offloading decision;

[0174] Step S4: Each node allocates computing resources to all tasks to be executed in the node according to a resource allocation method based on convex optimization to support their parallel operation;

[0175] Step S5: The system records the status of each time slot, the actions generated, the rewards obtained, and the new status entered. The DRL agent generates new service caching and computation offloading decisions based on the above information.

[0176] In this example, when the optimization method described in this example needs to be simulated, the method is:

[0177] The simulation experiments were conducted on a server equipped with an Intel Xeon Silver 4208 processor and an RTX 3090 graphics card. 2The area of ​​​​the proposed JCO-CR method is used as an experimental scenario. In this area, three BSs equipped with MEC servers are deployed with a coverage radius of 1km. 10 service types are considered, and their storage requirements are randomly distributed in the [2,5]GB range. The real-world Milan cellular traffic dataset is used to simulate the dynamically changing user requests in DTCEN. This dataset contains Milan user traffic data from November 1, 2013 to January 1, 2014, recorded every ten minutes, including events such as text messages, calls, and the Internet. The Internet traffic data of three of the areas are selected as the total number of task requests received by the BSs in each time slot. In real scenarios, user devices usually have a certain periodicity and regularity in their demand for different service types, so the experiment considers the popularity of service types. Specifically, the user device issues a task request at time slot t, and the popularity of the required service type follows the Zipf distribution. Let R k Indicates the service type k The popularity ranking of , then s k The probability of being requested can be expressed as:

[0178]

[0179] Where 0≤p k ≤1 and γ represents the Zipf bias factor. The larger γ is, the more service types required by user tasks are concentrated on service types with higher popularity rankings; conversely, when γ approaches 0, the service types required by user tasks are evenly distributed. Following the actual situation, γ is set to 0.8 in the experiment.

[0180] Based on Python 3.8, a DTCEN simulation scenario was built, and the proposed JCO-CR method was implemented using the open source machine learning framework Pytorch. In JCO-CR, the learning rate of the Actor network and the Critic network are both 1e-4, and the reward discount factor is 0.99. The other main simulation parameter settings are shown in Table 1. After training, JCO-CR can be applied to the joint optimization of service caching, computation offloading, and resource allocation in the dynamic and changeable DTCEN environment.

[0181] In this example, a comprehensive performance comparison of the proposed JCO-CR method is conducted with the following six baseline methods.

[0182] (1) DDPG: As an advanced DRL algorithm, DDPG combines deep neural networks (DNNs) and deterministic policy gradients to handle high-dimensional state spaces and continuous action spaces.

[0183] (2) TD3: As an advanced DRL algorithm, TD3 alleviates the problem of overestimation of Q value in DDPG by introducing dual Q network and delayed update, thus improving the stability of the algorithm.

[0184] (3) RC: Randomly caches services on BSs and adopts the same computation offloading and resource allocation strategy as JCO-CR.

[0185] (4) NCo: User tasks are executed in the local BS or CC without collaborative caching or collaborative offloading, and the same service caching strategy as JCO-CR is adopted.

[0186] (5) ACC: User tasks are executed on CC, and convex optimization theory is used to allocate the available computing resources of CC.

[0187] (6) ACA: User tasks are executed on CCs, evenly distributing the available computing resources of CCs.

[0188] The simulation parameters of the comparison process are as follows

[0189]

[0190] First, the convergence of different methods is compared. Figure 3 As shown in the figure, neither the ACC nor the ACA methods involve policy updates, so their performance will not change. The ACC method uses convex optimization theory for resource allocation, while the ACA method allocates resources evenly. Without considering service caching and computation offloading, the ACC method can obtain the optimal solution to the resource allocation subproblem, so its performance is higher than that of the ACA method. The RC method randomly caches services on BSs, reducing the cooperative cache hit rate between BSs, which causes more tasks to be offloaded to the CC for processing, thereby increasing service latency. The NCo method ignores cooperative caching, and user tasks can only be executed on local BSs or CCs. However, the NCo method uses the TD3 algorithm for service caching decisions, and the cache hit rate of local BSs will be improved as training progresses, so its performance is better than the RC method. Compared with the NCo method, the DDPG and TD3 methods improve the load balancing problem between BSs.

[0191] This is because these two methods realize collaborative caching of service types between BSs, and then realize collaborative offloading of user tasks (because task requests need to be offloaded to BSs that have cached the required service types before they can be executed). Compared with the DDPG method, the TD3 method achieved higher rewards. This is because TD3 uses a dual Q network and calculates the target Q value by selecting the smaller value of the dual Q network to reduce the overestimation bias of the Q value. Compared with other baseline methods, the proposed JCO-CR method combines convex optimization theory with the improved DRL method, effectively improving the stability of the training process and showing superior performance.

[0192] Next, the present invention evaluates the impact of BS cache capacity on the performance of different methods. Figure 4 As shown in the figure, the ACC and ACA methods forward all tasks to the CC for processing, which caches all service types. Therefore, the change in BS cache capacity has no effect on the performance of these two methods. The performance of the other five methods gradually improves with the increase of BS cache capacity. As the BS storage capacity continues to increase, it can cache more types of services to meet task requests. This improves the cooperative cache hit rate between BSs, so more tasks can be processed on the edge without uploading to the CC, which reduces service latency. In scenarios with different BS cache capacities, the JCO-CR method proposed in the present invention achieves the best performance, which verifies the effectiveness and superiority of the proposed method in solving the joint optimization problems of service caching, computation offloading and resource allocation. Specifically, the JCO-CR method achieves performance improvements of approximately 8.64%, 16.11%, 26.34%, 18.97%, 35.37% and 39.31% compared with the benchmark methods such as TD3, DDPG, RC, NCo, ACC and ACA, respectively.

[0193] Then, the impact of BS computing resources on the performance of different methods is compared. Figure 5As shown in the figure, the ACC and ACA methods forward tasks to the CC for processing, so the increase or decrease of BS computing resources will not affect their performance. With the continuous increase of BS computing resources, the other five methods all show different degrees of performance improvement. Compared with the TD3 method, the DDPG method has a slower upward trend. This is because the DDPG method has the problem of overestimation of Q value, which makes it easy to fall into the local optimum when selecting actions. In the scenario with different BS computing resources, the proposed JCO-CR method shows more excellent performance than other methods. This is because the JCO-CR method introduces convex optimization theory to accurately obtain the optimal resource allocation plan, which makes the JCO-CR method more advantageous in solving the joint optimization problems of service caching, computing offloading and resource allocation. Specifically, the JCO-CR method is compared with the benchmark methods such as TD3, DDPG, RC, NCo, ACC, and ACA, and has achieved performance improvements of about 14.63%, 24.08%, 32.29%, 23.81%, 42.61%, and 46.03%, respectively.

[0194] Finally, the performance gap between the JCO-CR method and other methods on different types of (communication and computation) delays is evaluated. Figure 6 As shown in the figure, the ACC and ACA methods upload tasks to the CC for execution, and the long distance between the terminal device and the CC causes excessive communication delay. However, because the CC has sufficient computing resources, the computational delay of these two methods is relatively low. The ACC method adopts resource allocation based on convex optimization, so its computational delay is reduced compared with the ACA method, showing higher computational efficiency. The RC method randomly caches service types, resulting in a low cache hit rate of BSs. In this case, many user tasks need to be offloaded to the CC for processing, resulting in high transmission delays. The delay strategy update mechanism and dual Q network introduced by the TD3 method effectively improve the defects of unstable training and over-estimation of Q values ​​in the DDPG method, thereby achieving better performance. Compared with other benchmark methods, the proposed JCO-CR method reasonably decouples the original optimization problem, and then designs a joint optimization strategy of service caching, computation offloading and resource allocation based on improved DRL and convex optimization theory, which effectively improves the performance of the method.

Claims

1. A method for optimizing service cache and computation offloading in a digital twin cloud-edge network, characterized in that: The method includes a virtual digital twin cloud-edge network model DTCEN, and also includes a service cache and computation offloading joint optimization method JCO-CR based on convex optimization enabled deep reinforcement learning. The DTCEN includes a system controller SyC, a cloud computing center CC, I base stations BSs and J user devices UDs; the BS set is denoted as B = {b1, b2, ..., b i ,…,b I }, the UD set is recorded as U = {u1,u2,…u j ,…u J Each BS is equipped with a MEC server for providing storage and computing resources, where b i The storage and computing resources are denoted as C i and F i UDs are randomly distributed within the communication coverage of each BS and take the BS with the nearest neighbor as the local BS; b i The number of UDs within the communication coverage is denoted as n i ,in There are K types of services, denoted as S = {s1, s2, ..., s k ,…,s K SyC is responsible for collecting system status information and transmitting it to the DRL agent, which then makes decisions on service caching and computation offloading.

2. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 1 is characterized in that: In the physical cloud-edge network corresponding to DTCEN, CC and BSs collect system operation status information and build a DTCEN model to achieve the mapping between the physical cloud-edge network and the virtual twin network, and continuously monitor the physical network status to ensure the consistency of the twin model with it; The DRL agent interacts with the DTCEN environment to continuously optimize the service cache and computing offloading strategy, and then feeds the learned model back to the physical cloud-edge network to improve training efficiency; When the dynamics of the capture system needs to be optimized, the system operation time is discretized into T equal-length time slots, denoted as T = {1, 2, ..., t, ..., T}; u j Generate a task request in each time slot, defined as a triplet task j = <d j ,cy j ,cs j >,j∈J, where d j Indicates the amount of data, cy j Indicates the calculation task j The number of CPU cycles required per bit of data, cs j Indicates the type of service that the task depends on; When task j Unloaded to b i When b i Cached tasks j Required service type cs j And can allocate sufficient computing resources to it, task j It can be executed smoothly; otherwise, task j Need to be offloaded to other BSs or CCs for execution.

3. The service cache and computation offloading optimization method under the digital twin cloud edge network according to claim 2 is characterized in that: If the UDs and CC communicate over long distances, and transferring all user tasks to the CC for execution may cause severe congestion in the core network and excessive service delays, the service cache and computation offloading strategies are optimized to minimize long-term service delays. The specific methods are as follows: Method 3.1, Cache model optimization, is used to enable the DRL agent to make appropriate service cache decisions in each time slot: At time slot t, the system service cache state is defined as Where m i,k ∈{0,1} represents b i The service cache status in; when m i,k =1, b i Cached service types k and occupies c k storage capacity; otherwise , service type k Not cached; b i The total capacity of the cache service type cannot exceed the size of its storage space. The cache capacity constraint is expressed as: Among them, c k Indicates the service type k The required cache capacity, C i Indicates b i The storage space limit; In this method, the service type cached in time slot t can be used to process the task request in time slot t+1, that is, the previous time slot b i The service cache decision is used as the service cache state of the next time slot; in DTCEN, each BS broadcasts cache information at the beginning of each time slot to achieve collaborative caching between BSs, thereby effectively reducing redundant service caches in the system; Method 3.2, communication model optimization, in this method, user equipment u j task j There are two ways to handle this: Method 1: j Offload to local BS for processing; Wireless communication is used between user equipment and local BS. When multiple user equipment send task requests at the same time, BS will use frequency division multiple access technology FDMA to allocate independent channels for them. According to Shannon's formula, u i With b i The transmission rate between is defined as: Among them, w j,i Indicates b i Assign to u i The channel bandwidth, p j Indicates u j Transmission power, g j,i Indicates u j With b i The channel gain between 2 represents Gaussian noise power; I j',i =Σp j' g j',i ,j'∈J / {j} represents the interference caused by other devices to the communication; therefore, the task upload delay is defined as: Method 2: If the required service type is not cached on the local BS, the task j Forwarded to cooperative BSs or CC for processing; when a task is generated, its input data needs to be uploaded from the user equipment to the local BS or forwarded; If task j Required service type cs j If the data is not cached in the local BS, it needs to be forwarded to the cooperative BSs or CC. Fiber communication is used between BSs and between BSs and CC. The number of communication hops between the local BS and the target BS is recorded as h. i,z , the length of a single-hop path is denoted as d i,z ; Let z represent the set of BSs and CCs, i.e. z∈I∪{c}; In addition, the communication transmission rate between BSs is denoted by R I,I , b i The communication transmission rate between CC is denoted as R I,c ; Therefore, the task forwarding delay is defined as: Where v represents the propagation velocity of light in a vacuum, ε represents the refractive index of the optical fiber, and the task communication delay is defined as: Method 3.3, calculation model optimization, specifically: do not consider some task offloading, that is, task j Can only be completely offloaded to one node for processing; task j The unloading decision is denoted as O j ∈I∪{0}, where O j =i means the task is offloaded to b i , O j = 0 means the task is offloaded to the remote CC; the user devices executing the task on node z are denoted as set X z ; To ensure the smooth execution of the task request, node z needs to cache the service type required by the task and allocate an appropriate amount of computing resources to the task; Since the computing resources equipped by a single BS are limited, there are the following resource constraints, which are expressed as: Among them, node z is assigned to u j The computing resource ratio is denoted as f z,j , the computational delay of a task is defined as: Among them, when z∈I, Indicates that the task is in b i The calculation performed on z = c, Indicates the computational delay of the task executed on the CC; the proportion of computing resources allocated to each task is subject to the following constraints: f min_z ≤f z,j ≤1,z∈I∪{c} Formula 8; Considering the communication and computation delays, task j The service delay is defined as:

4. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 3 is characterized in that: The method explores the optimal service cache, computation offloading and resource allocation strategy to minimize long-term service delay, which is formalized as an optimization problem and expressed in the following formula: Among them, X z represents the set of user devices that offload tasks to node z in time slot t; C1 represents the service cache decision; C2 represents the task offloading decision; C3 represents that the sum of the capacity of the service types cached by the BS cannot exceed its storage capacity upper limit; C4 represents the node z assigned to task j The computing resource ratio constraint; C5 means that the sum of the computing resource ratios allocated to tasks by node z cannot exceed 1; Considering the discrete decision space of service caching and computation offloading, the continuous decision space of resource allocation, and the nonlinear constraints in the optimization problem, P1 is the original optimization problem, which is regarded as a mixed integer nonlinear programming problem MINLP to solve.

5. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 4 is characterized in that: When solving problem P1, the system first performs model training and decision-making in DTCEN through the improved DRL method; then, in each time slot t, the service cache and computing offloading decisions are fed back to the physical cloud-edge network, and the task is offloaded to the target node; after determining the set of tasks that each node needs to process, the resource allocation plan for each node is further determined. Service cache, computing offloading and resource allocation belong to two different dimensions of problems in P1, which are solved in a serial manner. By decoupling P1, it is split into sub-problems P2 and P3 for separate solutions.

6. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 5 is characterized in that: The sub-problem P2 is specifically: to minimize the long-term service delay, jointly optimize the service cache and computation offloading strategy, which is expressed as: The sub-problem P3 is as follows: minimize the computational delay of node z by optimizing resource allocation; after determining the service cache and computation offloading strategies, the resource allocation process of each node is relatively independent; only the resource allocation strategy of a single node needs to be optimized independently, that is, in It indicates the task communication delay and is not related to the resource allocation process; Therefore, P3 is simplified to:

7. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 6 is characterized in that: The optimization method targets subproblem P2 and uses an improved DRL method to perform model training in DTCEN to learn the optimal service cache and computation offloading strategy, and then deploys the decision feedback to the physical cloud-edge network. That is, the exploration and learning efficiency of the DRL agent is improved by introducing DTCEN. Specifically: The DRL agent obtains the DTCEN environment state s at time slot t t , output action a t And get the reward signal r t , then go to the next state s t+1 , the process is expressed as MDP; specifically, the state space, action space and reward function are defined as follows: State space: state s t Contains task information from the user's device. t , BSs’ cache status information M t And the number of tasks N received by each BS in the current time slot t ; Therefore, the system state at time slot t is expressed as: s t ={Task t ,M t ,N t }, Formula XIII; Among them, task information Task t ={D t ,CY t ,CS t ,L t }, cache status information Number of tasks Action space: action a t Including service cache decision Cache t and task offloading decision Off t ; Therefore, the action at time slot t is expressed as: a t = {Cache t ,Off t },Formula 14; Among them, service cache decision Task offloading decision Reward function: Assuming the optimization goal is to minimize the long-term service delay, the reward function at time slot t is defined as: Wherein, J represents a set of user devices.

8. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 7 is characterized in that: In the optimization method, the Actor and Critic networks are trained by the proximal policy optimization method PPO to make optimal service caching and computation offloading decisions in the dynamically changing DTCEN; in PPO, the Actor network outputs actions under a given state and updates the policy network parameters by maximizing the expected total return; the Critic network evaluates the value of the policy and updates the evaluation network parameters by comparing the difference between the actual and predicted returns; The proximal strategy optimization method PRO is specifically as follows: First, initialize the Actor network π θ and Critic Network V ω and its network parameters θ and ω, the experience pool M, the number of training rounds E, the number of time slots T, and the number of training rounds epochs for each update; randomly initialize the service type cached by each BS until its storage capacity upper bound is reached; in each training round, the user equipment uses the nearest neighbor BS as the local BS and accesses the DTCEN through it; then, the system state s in time slot t is t Enter the Actor Network π θ , generating service cache and computation offloading decision a t , and then based on the cache decision Cache t Update system status, according to uninstall decision Off t The task is unloaded to the target node, where exploration noise ε is used to avoid falling into the local optimum. Then, each node uses convex optimization theory to derive the optimal resource allocation to minimize the task calculation delay. Then, the system feeds back the immediate reward r t And accumulate to the current round reward ep_re, and transfer to the next state s t+1 , and then the training samples (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool M; Next, N samples are sampled from M to calculate the timing difference error TD. The TD error is defined as: at l =r l +γV(s l+1 ,ω)-V(s l ,ω),l∈N Formula 16; Among them, V(s l+1 ,ω) and V(s l ,ω) respectively represent the Critic network parameter ω in state s l+1 and l The state value function under ; Then, the advantage function of N samples is calculated based on the TD error and target value Used for subsequent network parameter updates; the Actor network uses the old and new strategies to calculate the probability ratio p of a specific action under a given strategy t , and according to the objective function J CLIP (θ) calculates the loss function and then passes Update strategy; objective function J CLIP (θ) is expressed as: Critic network uses objective function J Value (w) Calculate the loss function and use gradient descent to update the network parameters ω, where J Value (w) is expressed as: in, is the actual state value of sample i, V(s i ,ω) is the state s i The state value estimated by the network parameter ω is obtained; by minimizing the mean square error loss, the Critic network parameter ω gradually improves its estimation accuracy of the state value; after each training episode, the samples in M ​​are cleared.

9. The method for optimizing service cache and computation offloading in a digital twin cloud-edge network according to claim 8 is characterized in that: After the DRL agent makes service caching and computation offloading decisions, all tasks in DTCEN have been assigned to each node to wait for execution; each node reasonably allocates its computing resources to ensure the smooth execution of tasks; that is, for the optimization problem P3, first prove that P3 is a convex optimization problem; then, based on the convex optimization resource allocation method, the optimal resource allocation for task requests in DTCEN is achieved under given caching and offloading decisions; The proof that P3 is a convex optimization problem is: It is necessary to prove that the objective function and all constraints of P3 are convex functions. From Formula 12, we can see that constraints C4 and C5 are affine functions and therefore convex functions. Next, we need to prove the convexity of the objective function. To simplify the description, constraint C4 is rewritten, and P3 can be transformed into: Among them, C6 and C7 represent the upper and lower bounds of the proportion of computing resources that can be allocated to a single task respectively; in order to prove that the objective function is a convex function, the Hessian matrix is ​​introduced, which describes the local curvature of the function; the calculation method of each element of the Hessian matrix of the objective function is as follows: Among them, C6 and C7 represent the upper and lower bounds of the proportion of computing resources that can be allocated to a single task respectively; in order to prove that the objective function is a convex function, the Hessian matrix is ​​introduced, which describes the local curvature of the function; the calculation method of each element of the Hessian matrix of the objective function is as follows: Among them, d j 、cy j 、F z They represent the data volume and computing density of the task and the computing resources of node z respectively, all of which are non-negative numbers. In addition, since 0<f min_z ≤f z,j ≤1, so Both have f z,j >0; According to the above analysis, it is proved that P3 is a convex optimization problem; then, the Lagrangian function is defined as follows: Among them, φ, γ j and j∈X z are the numbers of Lagrange multipliers corresponding to C4, C5 and C6 respectively; for a convex optimization problem, if a point x in the definition domain satisfies the KKT condition, that is, satisfies the gradient, complementary relaxation and constraint conditions, then the point is the global optimal solution; for P3, its KKT condition is expressed as follows: Where t∈T, j∈X z , z∈I∪{c}; from (b) and (c) γ j (f z,j -1)=0, we can get formula (e) and (f) γ j =0; then, we can derive formula g from formula a Substituting it into formula b, we can get formula h Finally, substituting formula h into formula g yields the optimal computing resource allocation solution set for node z: In each time slot t, node z can derive the optimal computing resource allocation for the task according to the above process, and then guide the DRL agent to further optimize the service cache and computing offloading strategy.

10. The method for optimizing service cache and computing offloading in a digital twin cloud-edge network according to claim 9 is characterized in that: The use process of the method comprises the following steps: Step S1, all user equipments in DTCEN access DTCEN through the local BS; Step S2: The DRL agent receives the task information from the user based on the relevant information of the user's device, including the task information sent by the user. t , BSs’ cache status information M t And the number of tasks N received by each BS in the current time slot t , to generate service caching and computation offloading decisions; Step S3: SyC updates the cache of the service entity set on each BS according to the generated service cache decision; SyC assigns each task to the corresponding node for execution based on the generated computation offloading decision; Step S4: Each node allocates computing resources to all tasks to be executed in the node according to a resource allocation method based on convex optimization to support their parallel operation; Step S5: The system records the status of each time slot, the actions generated, the rewards obtained, and the new status entered. The DRL agent generates new service caching and computation offloading decisions based on the above information.

Images