Digital twin-assisted edge network resource allocation optimization method

Abstract

The present invention discloses a digital twin-assisted edge network resource allocation optimization method, which belongs to the field of edge computing technology; including: establishing a digital twin-assisted edge network resource allocation system; determining the transmission delay and energy consumption required for computing tasks in offloading mode; constructing a multi-intelligence deep reinforcement learning model; training the multi-intelligence deep reinforcement learning model, and using the trained multi-intelligence deep reinforcement learning model to implement a resource allocation strategy. The present invention minimizes task processing delay and energy consumption by establishing a service cache module and a task offloading module; considers the state space of the MEC server and the user terminal device respectively, improves the task offloading success rate, significantly reduces the task processing delay and energy consumption, and further improves the accuracy of task offloading and resource allocation.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and specifically relates to a digital twin-assisted edge network resource allocation optimization method. Background Art

[0002] Intelligent offloading strategies for computing tasks are crucial for effectively utilizing computing resources in edge servers. As edge user terminal devices become more intelligent and advanced, optimal collaborative task offloading and service caching between user devices and edge servers can directly improve system performance.

[0003] Current research assumes that edge servers are capable of handling all types of computing tasks from mobile user terminal devices, whether it is image recognition, behavior monitoring, or any other type of user request service, and can provide the best service quality. However, the actual situation is that the storage resources of edge servers are limited, so it is actually unrealistic to expect edge servers to store all services at the same time. In view of this reality, it is necessary to carefully decide the service caching strategy of each edge server, that is, which software program codes and databases should be cached to correspond to the corresponding services. This decision will directly affect whether the mobile user terminal device chooses to offload tasks to the edge server, which will have a profound impact on the overall performance of the mobile edge computing (MEC) system. Therefore, in such a cache-assisted MEC system, it is particularly critical to ensure the effective coordination between service caching and task offloading.

[0004] Task offloading strategies and service caching are tightly coupled complex issues. The computing requirements of mobile user terminal devices, the resource availability of edge servers, and the wireless network environment are all time-varying and dynamic, and difficult to accurately predict in practice, which brings unprecedented challenges to cloud-edge-end intelligent collaboration.

[0005] In summary, how to solve the problem of task offloading and resource allocation in the case of decentralized resource allocation in edge networks, thereby improving the accuracy of task offloading and resource allocation is a technical problem that the present invention aims to solve. Summary of the invention

[0006] The purpose of the present invention is to provide a digital twin-assisted edge network resource allocation optimization method to solve the problems raised in the above background technology.

[0007] The object of the present invention is achieved by: a digital twin-assisted edge network resource allocation optimization method, characterized in that the method comprises the following steps:

[0008] Step S1: Establish a resource allocation system for the digital twin-assisted edge network;

[0009] The resource allocation system includes a network module, a service cache module, a task offloading module and a digital twin module;

[0010] Step S2: According to the established resource allocation system, determine the transmission delay and energy consumption required for the computing task in the offloading mode;

[0011] Step S3: construct a multi-intelligence deep reinforcement learning model;

[0012] The multi-intelligence deep reinforcement learning model includes a state space, an action space, and a reward space;

[0013] Step S4: training the multi-intelligence deep reinforcement learning model, and using the trained multi-intelligence deep reinforcement learning model to implement the resource allocation strategy;

[0014] The multi-intelligence deep reinforcement learning model is trained using the MASAC algorithm.

[0015] Preferably, the network module is used for communication between the core network, the edge server, and the user terminal device;

[0016] The service cache module is used to consider the frequency and size of service component requests, and utilize a dynamic service cache replacement mechanism to reasonably arrange the limited storage space of the edge server;

[0017] The task offloading module is used to make offloading decisions and calculate the computing delay and energy consumption required for the tasks generated by the user terminal device under the offloading decision;

[0018] The digital twin module maps the user terminal device and edge server physical entities into digital twins, considers time-varying channels, historical data, offloading decisions, and service cache decisions to build the edge server digital twin; and considers the mobile terminal location, task request, and transmission power to build the user terminal device digital twin.

[0019] Preferably, the network module includes a core network, an edge server and a user terminal device, and the core network is used To indicate that the edge server includes base stations and MEC servers, one base station is equipped with one MEC server. represents a base station;

[0020] The edge server has high-speed cache and computing capabilities, and Timely processing by user terminal equipment The resulting computationally intensive and latency-sensitive computing tasks

[0021] Both user terminal devices and edge servers are virtually connected to the digital twin through two-to-two communication to form a digital twin edge computing network. The digital twin edge computing network is used to simulate and optimize offloading strategies, realize low-latency virtual twins, and feed back the optimal strategy to the physical network for implementation.

[0022] Preferably, in step S2, the transmission delay and energy consumption required for the computing task in the offloading mode are determined according to the established resource allocation system, and the specific operations are:

[0023] Step S2-1: Obtain the computing task status generated by each terminal device in the corresponding MEC server in the edge network environment;

[0024] In each time slot, the computational tasks generated by the user terminal devices served by the base station are represented as an ordered vector Among them, c u Indicates the total computational effort of the task; d u Indicates the data size of the task, including program / database; Indicates the service type. For the service catalog, Indicates the processing delay tolerance of the task, that is, the total processing delay of the task does not exceed

[0025] When defining task offloading, define As including The set of task offloading decisions for all user terminal devices in is further described as

[0026] Among them, α u =1 or α u =0 determines whether to process the task locally; β u =1 or β u = 0 determines whether to offload the tasks of the user terminal equipment to the local base station for processing; γ u =1 or γ u =0 determines whether to offload the tasks of the user terminal equipment to the adjacent base station for processing; u =1 or δ u =0 determines whether to offload the tasks of the user terminal equipment to the core network for processing. Task offloading includes the above four offloading models;

[0027] For each computing task in time slot t, an offloading decision is selected, so the offloading decision needs to satisfy α u +β u +γ u +δ u =1,

[0028] Step S2-2: The digital twin module builds the digital twins of the MEC server and terminal devices;

[0029] The digital twin of the MEC server is:

[0030]

[0031] The digital twin of the terminal device:

[0032]

[0033] Among them, Φ is the mapping function of the digital twin, C n is the computing resource of the nth MEC server, F n is the communication resource of the nth MEC server, is the estimated task data size, To represent the estimated task computational effort, To represent the estimated service components required;

[0034] Step S2-3: The service cache module in the MEC server caches the service components of the corresponding computing tasks within the coverage range, and manages the storage space according to the dynamic service cache replacement mechanism;

[0035] The service components are represented as:

[0036]

[0037] Among them, Λ c is the deviation between the actual state and the estimated value of the task data size, Λ d is the deviation between the actual state and the estimated value of the task calculation, The deviation between the actual status and the estimated value of the service component required for the task; Δ c is the maximum range of the deviation between the actual state and the estimated value of the task data size, Δ d is the maximum range of deviation between the actual state and the estimated value of the task calculation quantity, The maximum range of deviation between the actual state and the estimated value of the service component required for the task;

[0038] Step S2-4: According to the service cache directory of the service cache module, the network module calculates the transmission time and energy consumption corresponding to the four offloading modes;

[0039] Step S2-5: According to the service cache directory of the service cache module, the task offloading module calculates the computing delay and energy consumption corresponding to the four offloading modes;

[0040] Step S2-6: Synchronously calculate the task processing status and form a mixed integer nonlinear optimization problem.

[0041] Preferably, in step S2-4, according to the service cache directory of the service cache module, the network module calculates the transmission time and energy consumption corresponding to the four offloading modes, specifically:

[0042] The network module calculates four offloading modes, including the time delay and energy consumption of tasks transmitted from user terminal equipment to local, the time delay and energy consumption of tasks transmitted from user terminal equipment to local base station, the time delay and energy consumption of tasks transmitted from user terminal equipment to adjacent edge servers, and the time delay and energy consumption of tasks transmitted from user terminal equipment to core network. There is no delay in the transmission of tasks from user terminal equipment to local.

[0043] The time delay and energy consumption of task transmission from user terminal equipment to local base station:

[0044] The total bandwidth of the base station is divided into M orthogonal channels, each channel has the same number of orthogonal subcarriers. Represents a collection of channels; user terminal equipment covered by the base station uses a u,m ∈{0, 1} represents the channel allocation decision of user terminal device u, indicating whether user terminal device u is in the channel Unload its tasks; a u,m =1 means that channel m is allocated to user terminal device u, otherwise a u,m =0;

[0045] Based on the Shannon capacity formula, the uplink transmission rate of user terminal device u on channel m in its local base station n is expressed as:

[0046]

[0047] Where B is the bandwidth of the wireless transmission channel, p u,m and h u They represent the transmission power of user terminal device u on channel m and the channel gain between user terminal device u and the base station, L u represents the distance between the user terminal device and the base station, μ = 4 is the defined path loss value; is the noise power; It is the reception interference from other user terminal devices to the base station on the channel. Among them, h u′ is the interference channel gain from the user terminal equipment to the base station on the channel; p u′,m′ is the transmission power of user terminal equipment u′ on channel m′; a u′,m′ To allocate channel m′ to user terminal equipment u′;

[0048] Each base station adopts the OFDMA scheme, and each user terminal device can be assigned at most one channel, so the following constraints need to be met:

[0049]

[0050] Each channel can only be assigned to one user terminal device at most, namely:

[0051]

[0052] Based on the above assumptions, the uplink transmission rate of the user terminal equipment in its local base station is expressed as:

[0053]

[0054] By sending task The resulting transmission delay is calculated as:

[0055]

[0056] The corresponding transmission energy consumption is:

[0057]

[0058] Among them, d u The amount of transmission task; is the uplink transmission rate;

[0059] The transmission delay and energy consumption of the task from the user terminal device to the adjacent edge server:

[0060] The transmission delay of the task from the user terminal device to the adjacent edge server is:

[0061]

[0062] The energy consumption of task transmission from user terminal device to adjacent edge server is:

[0063]

[0064] The transmission delay and energy consumption of the task from the user terminal device to the core network:

[0065] The transmission delay of the task from the user terminal device to the core network is:

[0066]

[0067] in, It is expressed as the data transmission rate of the wired connection from the base station to the cloud server; is the average transmission rate between the local base station n and the neighboring base station n′;

[0068] The transmission energy consumption of the task from the user terminal device to the core network is:

[0069]

[0070] Among them, Z n is the storage capacity of the base station, z k Is a cache service Required storage space, The service caching decision for service k in the base station at time slot t.

[0071] Preferably, in step S2-5, according to the service cache directory of the service cache module, the task offloading module calculates the computing delay and energy consumption corresponding to the four offloading modes, specifically:

[0072] The four offloading modes calculated by the task offloading module include the transmission delay and energy consumption of the task from the terminal device to the local processing, the transmission delay and energy consumption of the task from the terminal device to the local base station processing, the transmission delay and energy consumption of the task from the terminal device to the neighboring base station processing, and the transmission delay and energy consumption of the task from the terminal device to the core network processing;

[0073] The transmission delay and energy consumption of the task from the terminal device to the local processing: α u =1,β u =0,γ u =0,δ u =0;

[0074] When the service component corresponding to the task is cached in the terminal device, only local computing delay is generated, and there is no transmission delay. The computing delay expression is:

[0075]

[0076] Among them, f u is the local CPU cycle frequency;

[0077] The corresponding calculation energy consumption is:

[0078]

[0079] in, is the energy coefficient related to the chip architecture of the user terminal device;

[0080] The transmission delay and energy consumption of the task from the terminal device to the local base station: α u =0,β u =1,γ u =0,δ u =0;

[0081] The transmission delay of the task from the terminal device to the local base station is:

[0082]

[0083] Among them, f u Local BS-A The transmission frequency from the terminal device to the local base station processing;

[0084] The energy consumption of the task from the terminal device to the local base station is:

[0085]

[0086] Where ψ is the energy coefficient related to the chip architecture of the local base station;

[0087] The transmission delay and energy consumption of the task from the terminal device to the adjacent base station: α u =0,β u =0,γ u =1,δ u =0;

[0088] The transmission delay of the task from the terminal device to the adjacent base station processing:

[0089]

[0090] Among them, f u Cooperative BS-A The frequencies for transmissions handled from terminal devices to neighboring base stations;

[0091] The energy consumption of task transmission from terminal device to neighboring base station processing:

[0092]

[0093] Where ψ′ is the energy coefficient related to the chip architecture of the neighboring base station;

[0094] The core network has rich computing capabilities, and the transmission delay and energy consumption of tasks from terminal devices to core network processing are negligible.

[0095] Preferably, in step S2-6, the task processing status is synchronously calculated and a mixed integer nonlinear optimization problem is formed, and the specific operations are:

[0096] The mixed integer nonlinear optimization problem is to jointly minimize the task completion delay and energy consumption under resource constraints, make the optimal service cache and task offloading decisions, and determine the constraint target as:

[0097]

[0098] Among them, constraint C1 indicates that the task offloading decision is a binary variable, constraint C2 indicates that a task only selects one offloading mode, constraint C3 indicates that the cellular channel allocation decision is a binary variable, constraint C4 is expressed as an indicator of whether the service component is cached, constraint C5 is a binary indicator of whether the cached service component is requested, constraint C6 is a binary decision variable of whether the cached service component is replaced, and constraint C7 is expressed as a binary decision variable of whether the cached service component is replaced. The uniqueness of the offloading task decision, C8 represents the user terminal device Uniqueness of offloading task decision,Constraint C9 is to ensure that the total data size of the cached service cannot exceed the total storage of the edge server, and constraint C10 represents the guarantee that the processing delay of task u does not exceed the task processing delay tolerance; represents the caching decision, is the total task completion delay, is the total task completion energy consumption, is the service buffer decision of service k in the base station at time slot t, and represents the channel allocation decision of user terminal device u; Represents a computing task Whether to request service k, when When computing tasks Request service k; when Then service k is not available in the edge server.

[0099] Preferably, the state space is different for the two types of agents, the user terminal device and the MEC server. For the user terminal device, the information of the current computing task, the wireless channel state, and the transmission power are defined. For the MEC server, the computing resources and storage resources of the current MEC server are defined, which are expressed as follows:

[0100]

[0101] in, Indicates the status of the user terminal device u, Indicates the status of MEC server n; the status of each user terminal device u includes four aspects: Among them, c u represents the total computational effort of the task, d u Indicates the data size of the task, p u,m Indicates the transmit power, represents the interference between wireless channels;

[0102] The status of the MEC server includes two aspects: Among them, Z n represents the storage space of the MEC server, f uBS-A Indicates the computing capacity of the MEC server;

[0103] The action space includes two parts: MEC server cache action and user terminal device unloading action. The action space is expressed as follows:

[0104] in, represents the caching decision, Indicates uninstall decision;

[0105] The system reward function of the reward space is defined as:

[0106]

[0107] Among them, ζ is the constraint target weight coefficient; for total task completion delay; is the total energy consumption for completing the task.

[0108] Preferably, the MASAC algorithm includes an Actor network module and a Critic network, the Actor network module makes decisions based on local observations, and the Critic network uses global state and global action learning;

[0109] The MASAC algorithm introduces an additional constraint to maximize the policy output action association entropy, setting the state entropy to:

[0110]

[0111] in, is the expectation of action a, ε(π(·|s(t)) is the entropy of strategy π in state s(t); a(t) is the sampled action at time t; s(t) is the state at time t;

[0112] The objective function of the MASAC algorithm is:

[0113]

[0114] Where ο represents the discount factor and ρ represents the relative importance between reward and entropy;

[0115] The Soft Bellman equation is used to transform the objective function into:

[0116]

[0117] in, is the virtual state in the digital twin at time t+1, is the virtual decision in the digital twin at time t+1, a(t+1) is the sampling action at time t+1, s(t+1) is the state at time t+1, is the virtual state in the digital twin at time t, is the virtual decision in the digital twin at time t.

[0118] Preferably, in step S4, the multi-intelligence deep reinforcement learning model is trained, and the specific operations are:

[0119] Step S4-1: Sampling action samples: During the training process, the Actor network module first generates a Gaussian distribution and samples an action sample from the Gaussian distribution;

[0120] Step S4-2: Calculate the Actor network module update parameters:

[0121] The update rate of the Actor network module is to minimize L π (Ω):

[0122]

[0123] Where, ο is the discount factor; Q Ψ is the action value function when the update parameter is Ψ;

[0124] Step S4-3: Calculate the update parameters of the Critic network module and output the Q value; the update rate of the Critic network module is to minimize L Q (Ψ):

[0125]

[0126] in, The specific calculation method is:

[0127]

[0128] in, is the target critic network parameter, and ρ represents the relative importance parameter between reward and entropy.

[0129] Compared with the prior art, the present invention has the following improvements and advantages:

[0130] 1. By establishing a service cache module and a task offloading module, the task processing delay and energy consumption are minimized; the state space of the MEC server and the user terminal device are considered separately to improve the task offloading success rate, which significantly reduces the task processing delay and energy consumption.

[0131] 2. Through the twin model of the user terminal device, the actual task data size, task computing amount and required service component type are obtained, and the task completion delay and energy consumption are jointly minimized under resource constraints to make the optimal service caching and task offloading decisions; further improve the accuracy of task offloading and resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0132] Figure 1 Flow chart of the method of the present invention.

[0133] Figure 2 This is a structural diagram of the dynamic service cache replacement mechanism of the present invention.

[0134] Figure 3 This is a flow chart of the resource allocation optimization algorithm of the present invention.

[0135] Figure 4 This is a flow chart of determining the transmission delay and energy consumption required for computing tasks in the offload mode in the present invention.

[0136] Figure 5 The figure shows the system cost results of the offloading strategy under different numbers of terminal devices.

[0137] Figure 6 This is the result diagram of the task offloading success rate under different numbers of terminal devices of the offloading strategy. DETAILED DESCRIPTION

[0138] The present invention is further summarized below with reference to the accompanying drawings.

[0139] like Figure 1 As shown, a digital twin-assisted edge network resource allocation optimization method comprises the following steps:

[0140] Step S1: Establish a resource allocation system for the digital twin-assisted edge network;

[0141] The resource allocation system includes a network module, a service cache module, a task offloading module, and a digital twin module;

[0142] The network module includes a core network, an edge server, and a user terminal device, and is used for communication between the core network, the edge server, and the user terminal device;

[0143] Core network To indicate that the edge server includes base stations and MEC servers, one base station is equipped with one MEC server. represents a base station;

[0144] The edge server has high-speed cache and computing capabilities, and Timely processing by user terminal equipment The resulting computationally intensive and latency-sensitive computing tasks

[0145] Both user terminal devices and edge servers are virtually connected to the digital twin through two-to-two communication to form a digital twin edge computing network. The digital twin edge computing network is used to simulate and optimize offloading strategies, realize low-latency virtual twins, and feed back the optimal strategy to the physical network for implementation.

[0146] The service cache module is used to consider the request frequency and size of service components, and use the dynamic service cache replacement mechanism to reasonably arrange the limited storage space of the edge server;

[0147] like Figure 2 As shown, in the service cache module, the base station has cache and computing capabilities, which can cache services of computing-intensive and delay-sensitive applications and allow corresponding tasks to be processed on the edge network, thereby reducing the frequency of requests to the core network; not all services can be cached on the base station with limited storage capacity at the same time, so the base station needs to wisely decide which types of services should be cached;

[0148] Let binary variable is the service caching decision for service k in the base station at time slot t, where if service k is cached at the base station, then otherwise Considering that the total cached services cannot exceed the storage capacity of the base station, the following capacity constraints apply:

[0149]

[0150] Among them, Z n is the storage capacity of the base station, z k Is a cache service Required storage space.

[0151] In order to ensure effective task offloading, the dynamic service cache replacement mechanism is used to reasonably arrange the limited storage space of the edge server. The specific steps are as follows:

[0152] Step 1: Obtain the base station storage space currently occupied by the cached service component and record the current remaining storage space of the base station;

[0153] Step 2: For the new service component requested by the task, determine whether the remaining storage space of the base station can cache the new service based on the size of the new service component;

[0154] Step 3: If the remaining storage space of the base station is larger than the current service component size, the service component is directly cached;

[0155] Step 4: If the remaining storage space of the base station is less than the current service component size, the popularity of all cached service components is calculated and the cached service component list is sorted in descending order;

[0156] Step 5: traverse the service component list and compare the new service component request frequency with the service component request frequency cached in the base station;

[0157] Step 6: Remove the service components in the base station that have a request frequency lower than that of the new service components;

[0158] Step 7: Cache new service components and update the remaining storage space of the base station;

[0159] Step 8: If no suitable replacement service is found, the new service is not cached and the operation ends.

[0160] The task offloading module is used to make offloading decisions and calculate the computing delay and energy consumption required for the tasks generated by the user terminal device under the offloading decision;

[0161] The digital twin module maps user terminal devices and edge server physical entities into digital twins, considers time-varying channels, historical data, offloading decisions, and service caching decisions to build edge server digital twins; and considers mobile terminal locations, task requests, and transmission power to build user terminal device digital twins.

[0162] In step S2, according to the established resource allocation system, the transmission delay and energy consumption required for the computing task in the offloading mode are determined. The specific operations are:

[0163] like Figure 4 As shown, step S2-1: obtaining the computing task status generated by each user terminal device in the corresponding MEC server in the edge network environment;

[0164] In each time slot, the computational tasks generated by the user terminal devices served by the base station are represented as an ordered vector Among them, c u represents the total computational effort of the task, d u Indicates the data size of the task, including program / database; Indicates the service type. For the service catalog, Indicates the processing delay tolerance of the task, that is, the total processing delay of the task does not exceed

[0165] When defining task offloading, define As including The set of task offloading decisions for all user terminal devices in is further described as

[0166] Among them, α u =1 or α u =0 determines whether to process the task locally; β u =1 or β u= 0 determines whether to offload the tasks of the user terminal equipment to the local base station for processing; γ u =1 or γ u =0 determines whether to offload the tasks of the user terminal equipment to the adjacent base station for processing; u =1 or δ u =0 determines whether to offload the tasks of the user terminal equipment to the core network for processing. Task offloading includes the above four offloading models;

[0167] For each computing task in time slot t, an offloading decision is selected, so the offloading decision needs to satisfy α u +β u +γ u +δ u =1,

[0168] Step S2-2: The digital twin module builds the digital twins of the MEC server and the user terminal device;

[0169] The digital twin of the MEC server is:

[0170]

[0171] Digital twin of user terminal equipment:

[0172]

[0173] Among them, Φ is the mapping function of the digital twin, C n is the computing resource of the nth MEC server, F n is the communication resource of the nth MEC server, is the estimated task data size, To represent the estimated task computational effort, To represent the estimated service components required;

[0174] Step S2-3: The service cache module in the MEC server caches the service components of the corresponding computing tasks within the coverage range, and manages the storage space according to the dynamic service cache replacement mechanism;

[0175] The service components are represented as:

[0176]

[0177] Among them, Λ c is the deviation between the actual state and the estimated value of the task data size, Λ d is the deviation between the actual state and the estimated value of the task calculation, The deviation between the actual status and the estimated value of the service component required by the task; Δ cis the maximum range of the deviation between the actual state and the estimated value of the task data size, Δ d is the maximum range of deviation between the actual state and the estimated value of the task calculation quantity, The maximum range of deviation between the actual state and the estimated value of the service component required by the task.

[0178] Step S2-4: According to the service cache directory of the service cache module, the network module calculates the transmission time and energy consumption corresponding to the four offloading modes, specifically:

[0179] Four offloading modes are calculated under the network module, including the delay and energy consumption of tasks transmitted from user terminal devices to local, the delay and energy consumption of tasks transmitted from user terminal devices to local base stations, the delay and energy consumption of tasks transmitted from user terminal devices to adjacent edge servers, and the delay and energy consumption of tasks transmitted from user terminal devices to the core network. There is no delay in the transmission of tasks from user terminal devices to local.

[0180] The time delay and energy consumption of task transmission from user terminal equipment to local base station:

[0181] The total bandwidth of the base station is divided into M orthogonal channels, each channel has the same number of orthogonal subcarriers. Represents a collection of channels; user terminal equipment covered by the base station uses a u,m ∈{0, 1} represents the channel allocation decision of user terminal device u, indicating whether user terminal device u is in the channel Unload its tasks; a u,m =1 means that channel m is allocated to user terminal device u, otherwise a u,m =0;

[0182] Based on the Shannon capacity formula, the uplink transmission rate of user terminal device u on channel m in its local base station n is expressed as:

[0183]

[0184] Where B is the bandwidth of the wireless transmission channel, p u,m and h u They represent the transmission power of user terminal device u on channel m and the channel gain between user terminal device u and the base station, L u represents the distance between the user terminal device and the base station, μ = 4 is the defined path loss value; is the noise power; It is the reception interference from other user terminal devices to the base station on the channel. Among them, h u′ is the interference channel gain from the user terminal equipment to the base station on the channel; pu′,m′ is the transmission power of user terminal equipment u′ on channel m′; a u′,m′ To allocate channel m′ to user terminal equipment u′;

[0185] Each base station adopts the OFDMA scheme, and each user terminal device can be assigned at most one channel, so the following constraints need to be met:

[0186]

[0187] Each channel can only be assigned to one user terminal device at most, namely:

[0188]

[0189] Based on the above assumptions, the uplink transmission rate of the user terminal equipment in its local base station is expressed as:

[0190]

[0191] By sending task The resulting transmission delay is calculated as:

[0192]

[0193] The corresponding transmission energy consumption is:

[0194]

[0195] Among them, d u The amount of transmission task; is the uplink transmission rate;

[0196] The transmission delay and energy consumption of the task from the user terminal device to the adjacent edge server:

[0197] The transmission delay of the task from the user terminal device to the adjacent edge server is:

[0198]

[0199] The energy consumption of task transmission from user terminal device to adjacent edge server is:

[0200]

[0201] The transmission delay and energy consumption of the task from the user terminal device to the core network:

[0202] The transmission delay of the task from the user terminal device to the core network is:

[0203]

[0204] in, It is expressed as the data transmission rate of the wired connection from the base station to the cloud server; is the average transmission rate between the local base station n and the neighboring base station n′;

[0205] The transmission energy consumption of the task from the user terminal device to the core network is:

[0206]

[0207] Among them, Z n is the storage capacity of the base station, z k Is a cache service Required storage space, The service caching decision for service k in the base station at time slot t.

[0208] Step S2-5: According to the service cache directory of the service cache module, the task offloading module calculates the computing delay and energy consumption corresponding to the four offloading modes, specifically:

[0209] The transmission delay and energy consumption of the task from the terminal device to the local processing: α u =1,β u =0,γ u =0,δ u =0;

[0210] When the service component corresponding to the task is cached in the terminal device, only local computing delay is generated, and there is no transmission delay. The computing delay expression is:

[0211]

[0212] Among them, f u is the local CPU cycle frequency;

[0213] The corresponding calculation energy consumption is:

[0214]

[0215] in, is the energy coefficient related to the chip architecture of the user terminal device;

[0216] The transmission delay and energy consumption of the task from the terminal device to the local base station: α u =0,β u =1,γ u =0,δ u =0;

[0217] The transmission delay of the task from the terminal device to the local base station is:

[0218]

[0219] Among them, f u Local BS-A The transmission frequency from the terminal device to the local base station processing;

[0220] The energy consumption of the task from the terminal device to the local base station is:

[0221]

[0222] Where ψ is the energy coefficient related to the chip architecture of the local base station;

[0223] The transmission delay and energy consumption of the task from the terminal device to the adjacent base station: α u =0,β u =0,γ u =1,δ u =0;

[0224] The transmission delay of the task from the terminal device to the adjacent base station processing:

[0225]

[0226] Among them, f u Cooperative BS-A The frequencies for transmissions handled from terminal devices to neighboring base stations;

[0227] The energy consumption of task transmission from terminal device to neighboring base station processing:

[0228]

[0229] Where ψ′ is the energy coefficient related to the chip architecture of the neighboring base station;

[0230] When the service cache component does not exist in either the user terminal device or the MEC server, the task needs to be uploaded to the core network for processing. The core network has rich computing capabilities, and the transmission delay and energy consumption of the task from the user terminal device to the core network processing are negligible. Only the transmission delay and energy consumption are considered.

[0231] Step S2-6: Synchronously calculate the task processing status and form a mixed integer nonlinear optimization problem, specifically:

[0232] The mixed integer nonlinear optimization problem is to jointly minimize the task completion delay and energy consumption under resource constraints, make the optimal service cache and task offloading decisions, and determine the constraint objective as:

[0233]

[0234] Among them, constraint C1 indicates that the task offloading decision is a binary variable, constraint C2 indicates that a task only selects one offloading mode, constraint C3 indicates that the cellular channel allocation decision is a binary variable, constraint C4 is expressed as an indicator of whether the service component is cached, constraint C5 is a binary indicator of whether the cached service component is requested, constraint C6 is a binary decision variable of whether the cached service component is replaced, and constraint C7 is expressed as a binary decision variable of whether the cached service component is replaced. The uniqueness of the offloading task decision, C8 represents the user terminal device Uniqueness of offloading task decision,Constraint C9 is to ensure that the total data size of the cached service cannot exceed the total storage of the edge server, and constraint C10 represents the guarantee that the processing delay of task u does not exceed the task processing delay tolerance; represents the caching decision, is the total task completion delay, is the total task completion energy consumption, is the service buffer decision of service k in the base station at time slot t, and represents the channel allocation decision of user terminal device u; Represents a computing task Whether to request service k, when When computing tasks Request service k; when Then service k is not available in the edge server.

[0235] Step S3: construct a multi-intelligence deep reinforcement learning model;

[0236] The multi-intelligence deep reinforcement learning model includes state space, action space and reward space. The state space is different for the two types of agents, user terminal devices and MEC servers. For user terminal devices, the information of the current computing task, wireless channel status, and transmission power are defined. For MEC servers, the computing resources and storage resources of the current MEC server are defined, as shown below:

[0237]

[0238] in, Indicates the status of the user terminal device u, Indicates the status of MEC server n; the status of each user terminal device u includes four aspects: Among them, c u represents the total computational effort of the task, d u Indicates the data size of the task, p u,m Indicates the transmit power, represents the interference between wireless channels;

[0239] The status of the MEC server includes two aspects: Among them, Z n Indicates the storage space of the MEC server. Indicates the computing capacity of the MEC server;

[0240] The action space includes two parts: MEC server cache action and user terminal device unloading action. The action space is expressed as follows:

[0241] in, represents the caching decision, Indicates uninstall decision;

[0242] The system reward function in the reward space is defined as;

[0243]

[0244] Among them, ζ is the constraint target weight coefficient; for total task completion delay; is the total energy consumption for completing the task.

[0245] Step S4: training the multi-intelligence deep reinforcement learning model, and using the trained multi-intelligence deep reinforcement learning model to implement a resource allocation strategy;

[0246] like Figure 3 As shown in the figure, the multi-intelligence deep reinforcement learning model is trained using the MASAC algorithm. The MASAC algorithm includes an Actor network module and a Critic network. The Actor network module makes decisions based on local observations, and the Critic network uses global states and global actions for learning.

[0247] The MASAC algorithm introduces an additional constraint to maximize the policy output action correlation entropy, setting the state entropy to:

[0248]

[0249] in, is the expectation of action a, ε(π(·|s(t)) is the entropy of strategy π in state s(t); a(t) is the sampled action at time t; s(t) is the state at time t;

[0250] The objective function of the MASAC algorithm is:

[0251]

[0252] Where ο represents the discount factor and ρ represents the relative importance between reward and entropy;

[0253] The Soft Bellman equation is used to transform the objective function into:

[0254]

[0255] in, is the virtual state in the digital twin at time t+1, is the virtual decision in the digital twin at time t+1, a(t+1) is the sampling action at time t+1, s(t+1) is the state at time t+1, is the virtual state in the digital twin at time t, is the virtual decision in the digital twin at time t.

[0256] To train the multi-intelligent deep reinforcement learning model, the specific operations are as follows:

[0257] Step S4-1: Sampling action samples: During the training process, the Actor network module first generates a Gaussian distribution and samples an action sample from the Gaussian distribution;

[0258] Step S4-2: Calculate the Actor network module update parameters:

[0259] The update rate of the Actor network module is to minimize L π (Ω):

[0260]

[0261] Where, ο is the discount factor; Q Ψ is the action value function when the update parameter is Ψ;

[0262] Step S4-3: Calculate the update parameters of the Critic network module and output the Q value; the update rate of the Critic network module is to minimize L Q (Ψ):

[0263]

[0264] in, The specific calculation method is:

[0265]

[0266] Among them, Ψ is the target critic network parameter, and ρ represents the relative importance parameter between reward and entropy.

[0267] In order to verify the feasibility and effect of the present invention, the method of the present invention was tested:

[0268] Experimental environment: Consider a digital twin-assisted MEC solution, which includes 2 edge servers, 1 cloud server and 20 terminal devices; the coverage radius of the edge server is 200m, the terminal devices are scattered in different edge server coverage areas, and the distance between two terminal devices is evenly distributed in [1m, 50m]; the edge server is equipped with multiple CPU processors with a total computing power of 20GHz; in addition, it is assumed that the computing power of the terminal devices is evenly distributed in [0.9GHz, 1.5GHz], and the battery level initially assigned to each terminal device is evenly distributed in [30%, 100%]; since the cloud server is equipped with a high-speed multi-core CPU processor, the computing power of the cloud is much greater than that of the edge server, and the computing delay of the cloud can be ignored.

[0269] like Figure 5 As shown in the figure, deep Q learning implements a DQN-based offloading method for the collaborative mobile edge computing system; the discount factor and the size of the replay buffer are set to the same parameters as the algorithm proposed in this paper; local computing means that all tasks are executed locally on the terminal device; edge-only computing means that all tasks are executed on the local MEC server;

[0270] The no offloading solution means that the MEC server in this solution only acts as a relay node and has no computing power, and all tasks are processed by the cloud server;

[0271] The no-digital twin solution means that in this solution, DT is not used; without considering the system status, the user terminal device randomly assigns tasks to the local MEC server or cloud server.

[0272] It can be found that as the number of user terminal devices increases, more computing tasks will be generated and processed, so the total system cost will continue to increase; at the same time, the total system cost of the method proposed in this invention is the lowest; when the number of user terminal devices is 30, the method of this invention can reduce the system cost by about 42.5% compared with the local computing method; this is because the MASAC method enables the MEC server to cache more appropriate related services and enables the user terminal device to select a more appropriate computing offloading strategy. However, the deep Q learning method does not consider service caching, so its task overhead is slightly higher than the method proposed in this article.

[0273] like Figure 6As shown in the figure, the offloading success rate is defined as the ratio of the number of successfully processed tasks to the total number of tasks, which can measure the instantaneous performance of a single offloading task; as the number of user terminal devices increases, the offloading success rate shows a downward trend; this is mainly because the scale of computing tasks grows, resulting in higher demand for computing resources; however, the storage space and computing power of the MEC server remain fixed, and because the user terminal devices need to use limited storage resources and computing resources between them, the task eventually fails; in contrast, the offloading scheme proposed in the present invention effectively solves the problem of resource limitation by utilizing cloud-edge-end collaboration to process tasks; therefore, compared with the rest of the methods, this method significantly improves the task offloading rate by 45.8%. It is worth noting that the local computing and non-offloading schemes show certain fluctuations; this is because the task completion rate of local computing depends entirely on the computing power of the user terminal device and the size of the generated tasks; for the non-offloading scheme, the computing resources are sufficient, but are limited by the bandwidth of the wireless link.

[0274] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A digital twin-assisted edge network resource allocation optimization method, characterized in that: The method comprises the following steps: Step S1: Establish a resource allocation system for the digital twin-assisted edge network; The resource allocation system includes a network module, a service cache module, a task offloading module and a digital twin module; Step S2: Determine the transmission delay and energy consumption required for the computing task in the offloading mode according to the established resource allocation system; Step S3: construct a multi-intelligence deep reinforcement learning model; The multi-intelligent deep reinforcement learning model includes a state space, an action space, and a reward space; the state space is different for the two types of intelligent agents, the user terminal device and the MEC server. For the user terminal device, the information of the current computing task, the wireless channel state, and the transmission power are defined. For the MEC server, the computing resources and storage resources of the current MEC server are defined, which are expressed as follows: in, Indicates the status of the user terminal device u, Indicates the status of MEC server n; the status of each user terminal device u includes four aspects: Among them, c u represents the total computational effort of the task, d u Indicates the data size of the task, p u,m Indicates the transmit power, represents the interference between wireless channels; The status of the MEC server includes two aspects: Among them, Z n Indicates the storage space of the MEC server. Indicates the computing capacity of the MEC server; The action space includes two parts: MEC server cache action and user terminal device unloading action. The action space is expressed as follows: in, represents the caching decision, Indicates uninstall decision; The system reward function of the reward space is defined as: Among them, ζ is the constraint target weight coefficient; for total task completion delay; The energy consumption for the total task completion; Step S4: training the multi-intelligence deep reinforcement learning model, and using the trained multi-intelligence deep reinforcement learning model to implement a resource allocation strategy; The multi-intelligence deep reinforcement learning model is trained using the MASAC algorithm.

2. The digital twin-assisted edge network resource allocation optimization method according to claim 1 is characterized in that: The network module is used for communication between the core network, edge server and user terminal equipment; The service cache module is used to consider the frequency and size of service component requests, and utilize a dynamic service cache replacement mechanism to reasonably arrange the limited storage space of the edge server; The task offloading module is used to make offloading decisions and calculate the computing delay and energy consumption required for the tasks generated by the terminal device under the offloading decision; The digital twin module maps the terminal device and edge server physical entities into digital twins, considers time-varying channels, historical data, offloading decisions, and service cache decisions to build the edge server digital twin; and considers the mobile terminal location, task request, and transmission power to build the terminal device digital twin.

3. The digital twin-assisted edge network resource allocation optimization method according to claim 2 is characterized in that: The network module includes a core network, an edge server and a user terminal device. To indicate that the edge server includes base stations and MEC servers, one base station is equipped with one MEC server. represents a base station; The edge server has high-speed cache and computing capabilities, and Timely processing by user terminal equipment The resulting computationally intensive and latency-sensitive computing tasks Both user terminal devices and edge servers are virtually connected to the digital twin through two-to-two communication to form a digital twin edge computing network. The digital twin edge computing network is used to simulate and optimize offloading strategies, realize low-latency virtual twins, and feed back the optimal strategy to the physical network for implementation.

4. The digital twin-assisted edge network resource allocation optimization method according to claim 1, characterized in that: In step S2, the transmission delay and energy consumption required for the computing task in the offloading mode are determined according to the established resource allocation system. The specific operations are: Step S2-1: Obtain the computing task status generated by each terminal device in the corresponding MEC server in the edge network environment; In each time slot, the computational tasks generated by the user terminal devices served by the base station are represented as an ordered vector Among them, c u Indicates the total computational effort of the task; d u Indicates the data size of the task, including program / database; Indicates the service type. For the service catalog, Indicates the processing delay tolerance of the task, that is, the total processing delay of the task does not exceed When defining task offloading, define As including The set of task offloading decisions for all user terminal devices in is further described as Among them, α u =1 or α u =0 determines whether to process the task locally; β u =1 or β u = 0 determines whether to offload the tasks of the user terminal equipment to the local base station for processing; γ u =1 or γ u =0 determines whether to offload the tasks of the user terminal equipment to the adjacent base station for processing; u =1 or δ u =0 determines whether to offload the tasks of the user terminal equipment to the core network for processing. Task offloading includes the above four offloading models; For each computing task in time slot t, an offloading decision is selected, so the offloading decision must satisfy Step S2-2: The digital twin module builds the digital twins of the MEC server and terminal devices; The digital twin of the MEC server is: The digital twin of the terminal device: Among them, Φ is the mapping function of the digital twin, C n is the computing resource of the nth MEC server, F n is the communication resource of the nth MEC server, is the estimated task data size, To represent the estimated task computational effort, To represent the estimated service components required; Step S2-3: The service cache module in the MEC server caches the service components of the corresponding computing tasks within the coverage range, and manages the storage space according to the dynamic service cache replacement mechanism; The service components are represented as: Among them, Λ c is the deviation between the actual state and the estimated value of the task data size, Λ d is the deviation between the actual state and the estimated value of the task calculation, The deviation between the actual status and the estimated value of the service component required for the task; Δ c is the maximum range of deviation between the actual state and the estimated value of the task data size, Δ d is the maximum range of deviation between the actual state and the estimated value of the task calculation quantity, The maximum range of deviation between the actual state and the estimated value of the service component required for the task; Step S2-4: According to the service cache directory of the service cache module, the network module calculates the transmission time and energy consumption corresponding to the four offloading modes; Step S2-5: According to the service cache directory of the service cache module, the task offloading module calculates the computing delay and energy consumption corresponding to the four offloading modes; Step S2-6: Synchronously calculate the task processing status and form a mixed integer nonlinear optimization problem.

5. The digital twin-assisted edge network resource allocation optimization method according to claim 4 is characterized in that: In step S2-4, the network module calculates the transmission time and energy consumption corresponding to the four offloading modes according to the service cache directory of the service cache module, specifically: The network module calculates four offloading modes, including the time delay and energy consumption of tasks transmitted from user terminal equipment to local, the time delay and energy consumption of tasks transmitted from user terminal equipment to local base station, the time delay and energy consumption of tasks transmitted from user terminal equipment to adjacent edge servers, and the time delay and energy consumption of tasks transmitted from user terminal equipment to core network. There is no delay in the transmission of tasks from user terminal equipment to local. The time delay and energy consumption of task transmission from user terminal equipment to local base station: The total bandwidth of the base station is divided into M orthogonal channels, each channel has the same number of orthogonal subcarriers. represents a collection of channels; User terminal equipment covered by the base station uses a u,m ∈{0, 1} represents the channel allocation decision of user terminal device u, indicating whether user terminal device u is in the channel Unload its tasks; a u,m =1 means that channel m is allocated to user terminal device u, otherwise a u,m =0; Based on the Shannon capacity formula, the uplink transmission rate of user terminal device u on channel m in its local base station n is expressed as: Where B is the bandwidth of the wireless transmission channel, p u,m and h u They represent the transmission power of user terminal device u on channel m and the channel gain between user terminal device u and the base station, L u represents the distance between the user terminal device and the base station, μ = 4 is the defined path loss value; is the noise power; It is the reception interference from other user terminal devices to the base station on the channel. Among them, h u′ is the interference channel gain from the user terminal equipment to the base station on the channel; p u′,m′ is the transmission power of user terminal equipment u′ on channel m′; a u′,m′ To allocate channel m′ to user terminal equipment u′; Each base station adopts the OFDMA scheme, and each user terminal device can only be assigned one channel at most, so the following constraints need to be met: Each channel can only be assigned to one user terminal device at most, namely: Based on the above assumptions, the uplink transmission rate of the user terminal device in its local base station is expressed as: By sending task The resulting transmission delay is calculated as: The corresponding transmission energy consumption is: Among them, d u The amount of transmission task; is the uplink transmission rate; The transmission delay and energy consumption of the task from the user terminal device to the adjacent edge server: The transmission delay of the task from the user terminal device to the adjacent edge server is: The energy consumption of task transmission from user terminal device to adjacent edge server is: The transmission delay and energy consumption of the task from the user terminal device to the core network: The transmission delay of the task from the user terminal device to the core network is: in, It is expressed as the data transmission rate of the wired connection from the base station to the cloud server; is the average transmission rate between the local base station n and the neighboring base station n′; The transmission energy consumption of the task from the user terminal device to the core network is: Among them, Z n is the storage capacity of the base station, z k Is a cache service Required storage space, The service caching decision for service k in the base station at time slot t.

6. The digital twin-assisted edge network resource allocation optimization method according to claim 4 is characterized in that: In step S2-5, the task offloading module calculates the computing delay and energy consumption corresponding to the four offloading modes according to the service cache directory of the service cache module, specifically: The four offloading modes calculated by the task offloading module include the transmission delay and energy consumption of the task from the terminal device to the local processing, the transmission delay and energy consumption of the task from the terminal device to the local base station processing, the transmission delay and energy consumption of the task from the terminal device to the neighboring base station processing, and the transmission delay and energy consumption of the task from the terminal device to the core network processing; The transmission delay and energy consumption of the task from the terminal device to the local processing: α u =1,β u =0,γ u =0,δ u =0; When the service component corresponding to the task is cached in the terminal device, only local computing delay is generated, and there is no transmission delay. The computing delay expression is: Among them, f u is the local CPU cycle frequency; The corresponding calculation energy consumption is: in, is the energy coefficient related to the chip architecture of the user terminal device; The transmission delay and energy consumption of the task from the terminal device to the local base station: α u =0,β u =1,γ u =0,δ u =0; The transmission delay of the task from the terminal device to the local base station is: in, The transmission frequency from the terminal device to the local base station processing; The energy consumption of the task from the terminal device to the local base station is: Where ψ is the energy coefficient related to the chip architecture of the local base station; The transmission delay and energy consumption of the task from the terminal device to the adjacent base station: α u =0,β u =0,γ u =1,δ u =0; The transmission delay of the task from the terminal device to the adjacent base station processing: in, The frequencies for transmissions handled from terminal devices to neighboring base stations; The energy consumption of task transmission from terminal device to neighboring base station processing: Where ψ′ is the energy coefficient related to the chip architecture of the neighboring base station; The core network has rich computing capabilities, and the transmission delay and energy consumption of tasks from terminal devices to core network processing are negligible.

7. The digital twin-assisted edge network resource allocation optimization method according to claim 5 is characterized in that: In step S2-6, the task processing status is synchronously calculated and a mixed integer nonlinear optimization problem is formed. The specific operations are: The mixed integer nonlinear optimization problem is to jointly minimize the task completion delay and energy consumption under resource constraints, make the optimal service cache and task offloading decisions, and determine the constraint target as: Among them, constraint C1 indicates that the task offloading decision is a binary variable, constraint C2 indicates that a task only selects one offloading mode, constraint C3 indicates that the cellular channel allocation decision is a binary variable, constraint C4 is expressed as an indicator of whether the service component is cached, constraint C5 is a binary indicator of whether the cached service component is requested, constraint C6 is a binary decision variable of whether the cached service component is replaced, and constraint C7 is expressed as a binary decision variable of whether the cached service component is replaced. The uniqueness of the offloading task decision, C8 represents the user terminal device Uniqueness of offloading task decision,Constraint C9 is to ensure that the total data size of the cached service cannot exceed the total storage of the edge server, and constraint C10 represents the guarantee that the processing delay of task u does not exceed the task processing delay tolerance; represents the caching decision, is the total task completion delay, is the total task completion energy consumption, is the service buffer decision of service k in the base station at time slot t, and represents the channel allocation decision of user terminal device u; Represents a computing task Whether to request service k, when When computing tasks Request service k; when Then service k is not available in the edge server; ζ is the constraint target weight coefficient.

8. The digital twin-assisted edge network resource allocation optimization method according to claim 1, characterized in that: The MASAC algorithm includes an Actor network module and a Critic network. The Actor network module makes decisions based on local observations, and the Critic network uses global states and global actions for learning. The MASAC algorithm introduces an additional constraint to maximize the policy output action association entropy, setting the state entropy to: in, is the expectation of action a, ε(π(·|s(t)) is the entropy of strategy π in state s(t); a(t) is the sampled action at time t; s(t) is the state at time t; The objective function of the MASAC algorithm is: Where ο represents the discount factor and ρ represents the relative importance between reward and entropy; The Soft Bellman equation is used to transform the objective function into: in, is the virtual state in the digital twin at time t+1, is the virtual decision in the digital twin at time t+1, a(t+1) is the sampling action at time t+1, s(t+1) is the state at time t+1, is the virtual state in the digital twin at time t, is the virtual decision in the digital twin at time t; is the system reward function in the reward space.

9. The digital twin-assisted edge network resource allocation optimization method according to claim 1, characterized in that: In step S4, the multi-intelligent deep reinforcement learning model is trained, and the specific operations are as follows: Step S4-1: Sampling action samples: During the training process, the Actor network module first generates a Gaussian distribution and samples an action sample from the Gaussian distribution; Step S4-2: Calculate the Actor network module update parameters: The update rate of the Actor network module is to minimize L π (Ω): Where, ο is the discount factor; Q Ψ is the action value function when the update parameter is Ψ; s(t) is the state at time t; a(t) is the sampled action at time t; Step S4-3: Calculate the update parameters of the Critic network module and output the Q value; the update rate of the Critic network module is to minimize L Q (Ψ): in, The specific calculation method is: in, is the target Critic network parameter, ρ represents the relative importance parameter between reward and entropy; is the virtual state in the digital twin at time t, is the virtual decision in the digital twin at time t.

Images