Calculation unloading scheme based on digital twinning dynamic updating

Abstract

The invention provides a calculation unloading scheme based on digital twinning dynamic updating, and belongs to the field of mobile edge calculation. The method comprises the following steps: acquiring state information of a mobile device, a task and a base station in a current time slot; establishing a communication model, a task calculation model and an energy consumption model; establishing a comprehensive index function fusing digital twinborn precision, task delay and system energy consumption; and the comprehensive index functions with continuous time are summed to serve as a long-term dynamic optimization problem. In order to solve the problem, the invention further provides a hierarchical hybrid strategy deep reinforcement learning framework, and the framework combines a strategy learning mechanism and a non-strategy learning mechanism so as to jointly optimize a calculation unloading decision, a resource allocation strategy and an updating process of a digital twinborn model. The method aims at improving the precision of the digital twinborn body, remarkably reduces task delay and system energy consumption at the same time, and can support strategy generation in highly dynamic scenes such as smart cities due to the advantages of low calculation complexity and fast decision response.

Description

Technical Field

[0001] This invention belongs to the field of IoT edge computing technology, and particularly relates to a computing offloading scheme based on dynamic updates of digital twins. Background Technology

[0002] With the widespread adoption of IoT applications, edge computing has become a key technology for meeting the low-latency, high-quality service demands of mobile devices. However, user mobility leads to a highly dynamic edge network environment, which traditional static optimization methods struggle to address. Therefore, digital twin technology has been introduced to construct virtual models of physical systems, enabling accurate mapping and forward-looking decision-making in dynamic environments. Currently, the effectiveness of digital twin models heavily relies on their update frequency, but existing research often employs real-time or fixed-period updates. This approach presents a fundamental contradiction: while overly frequent updates ensure model accuracy, they consume significant amounts of scarce bandwidth and energy resources on the edge, leading to a decrease in task offloading completion rates; conversely, excessively low update frequencies result in outdated model information and inaccurate task offloading. Therefore, achieving a dynamic balance between system task offloading and model accuracy is a core problem that urgently needs to be solved in the application of digital twins in edge computing.

[0003] Furthermore, in dynamically changing edge environments, obtaining the optimal update strategy and task offloading scheme is a complex time-series decision-making problem, and traditional fixed strategies lack adaptability and foresight. Although deep reinforcement learning offers new ideas for solving such complex decision-making problems, designing a dynamic decision-making mechanism that can sense the real-time state of the system and intelligently adjust the update frequency and task offloading accordingly, while ensuring the efficiency and convergence of the learning process, remains a challenge. Therefore, the industry urgently needs a computational offloading scheme based on digital twin dynamic updates that can efficiently learn and adaptively make decisions, in order to maximize the resource utilization efficiency of edge computing systems while ensuring decision accuracy. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a computational offloading scheme based on dynamic updates of digital twins. The technical problem it aims to solve is that existing digital twin-assisted edge computing solutions suffer from the following deficiencies: First, existing methods mostly employ real-time or fixed-period update strategies, lacking adaptability and failing to effectively balance the accuracy of the digital twin model with the network resource and equipment energy consumption costs incurred by updates, adapting to dynamically changing system states and task requirements. Second, existing task offloading and resource allocation strategies are typically decoupled from the digital twin update process, performing independent or step-by-step optimizations, resulting in a lack of synergy between sub-problems and difficulty in achieving globally optimal overall system performance. Third, the dynamic and complex nature of edge computing environments leads to high-dimensional optimization problems and long decision chains. Traditional optimization methods or conventional deep reinforcement learning algorithms often suffer from high computational complexity, slow convergence speed, or poor decision-making performance when solving such problems.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a computational offloading scheme based on dynamic updates of digital twins, comprising the following steps:

[0006] S1. Obtain the location information of the mobile device in the current time slot, the task information generated by the device, the energy consumption status of the device, and the available resource status information of the base station;

[0007] S2. Based on the real-time status information obtained in step S1, establish a communication model to describe the data transmission process and a task computing model to describe the task processing flow, including a digital twin layer computing model, a local computing model, and an edge computing model; at the same time, construct an energy consumption model for the device based on the computing model.

[0008] S3. Construct an optimization function aimed at improving the overall performance of the system. This function integrates three core performance indicators: the accuracy of the digital twin, the task execution latency, and the energy consumption.

[0009] S4. By defining the multi-objective optimization function described in step S3 for multiple consecutive time slots, and summing the functions of all time slots over a long period of time, the final long-term dynamic optimization problem is constructed. The optimization objective of this problem is to maximize the accuracy of the digital twin and minimize the weighted sum of task execution delay and system energy consumption.

[0010] S5. Based on the final optimization problem, a hierarchical hybrid strategy deep reinforcement learning framework is used to solve the final optimization problem constructed in step S4. The framework combines policy and non-policy learning mechanisms to jointly optimize computational offloading decisions, resource allocation strategies, and the update frequency of digital twins, ultimately obtaining the optimal task offloading scheme.

[0011] The beneficial effects of this invention are as follows: By constructing a multi-objective optimization problem that integrates model accuracy, task latency, and system energy consumption, this invention can adaptively adjust the update strategy of the digital twin according to the real-time state of the system. While ensuring the accuracy of the twin, it effectively reduces task execution latency and system energy consumption, solving the problems of resource waste or insufficient accuracy caused by fixed update strategies. The hierarchical hybrid strategy deep reinforcement learning framework proposed in this invention can jointly optimize high-level digital twin update decisions with low-level computational offloading and resource allocation decisions, realizing the synergy of multi-dimensional strategies and thus obtaining better overall system performance. The asynchronous two-layer learning framework adopted by this method effectively reduces the computational complexity of solving long-term dynamic optimization problems, has the advantages of fast decision response and good convergence performance, and can efficiently support task processing and dynamic generation of transmission strategies in highly dynamic scenarios such as smart cities.

[0012] Furthermore, the expression for the communication model is as follows: Among them, h mn (t) represents the channel gain between device m and base station n at time slot t, R mn (t) represents the communication rate between base station n and device m located within the coverage area of ​​base station n in time slot t, v c A represents the speed of light. d f represents the antenna gain. c For the carrier frequency, d e Indicates path loss index, The distance between base station n and device m at time slot t is represented by B, where B is the wireless channel bandwidth, and p is the Euclidean distance between them. m Let m represent the transmit power of device m, N0 represent the Gaussian noise power spectral density, and I represent the intra-cell interference between devices transmitting to the same base station simultaneously.

[0013] Furthermore, the expression for the task computation model of the digital twin layer is as follows: in, This represents the total amount of update data that the device needs to upload in the first time slot t of each Δ time slot. This represents the upload delay of digital twin information between base station n and device m in the first time slot t of each Δ time slot. This indicates the edge computing latency within the digital twin layer. α represents the local computation latency within the digital twin layer. f The data size per unit of computing resource (e.g., 0.5KB / CPU) is quantified, α o This represents the amount of data requested for each task unloading request (e.g., 2KB / request), α ι The location data precision was simulated (e.g., 10 bytes / meter). This represents the available computing resources (O) of device m. mn (t) represents a summary of the computation task, ι m (t)=(x m (t),y m (t) represents the position of device m at the beginning of time slot t, Δ represents the number of time slots in the digital twin update cycle, and D mn (t) represents the data size of the task, C mn (t) represents the CPU cycles required to process one bit of data, f mn (t) and f m (t) represents the task O assigned by the digital twin layer guidance base station n and device m in time slot t. mn The computational resources of (t).

[0014] Furthermore, the expressions for the task-local computing model and the edge computing model are as follows: in, This represents the communication delay between base station n and mobile device m in time slot t. Indicates the actual edge computing latency. This represents the actual local computation delay, where ú{x} is an indicator function, ú{x} = 1 when the condition x is true, otherwise ú{x} = 0; This represents the estimated deviation between the resource allocation of the digital twin layer and the actual resource allocation, where β represents a constant of the slope of an adjustment function F(Δ).

[0015] Furthermore, the energy consumption model for physical layer devices is expressed as follows: in, This represents the communication energy consumption of mobile device m in time slot t. p represents the calculated energy consumption of device m in time slot t. m Let κ represent the transmit power of the mobile device m, and let κ represent a constant of the effective CPU switching capacitor.

[0016] Furthermore, the expression for the comprehensive performance index of the device is as follows: Among them, T mn (t) represents the total execution delay of the task, E m (t) represents the total energy consumption of device m, Φ m (t) represents the overall performance index. Indicates the bias of a digital twin, describing its accuracy; x mn (t)∈{0,1} represents a binary unloaded variable, x mn (t) = 1 indicates that task O mn(t) is offloaded to base station n for execution in time slot t, while x mn (t) = 0 indicates that task O mn (t) is calculated locally.

[0017] The beneficial effect of the above-mentioned further solutions is that the present invention constructs a comprehensive index Φ that integrates the accuracy, task execution latency, and system energy consumption of digital twins. m (t). Specifically, this function quantifies three core performance metrics that are interrelated and often conflicting in practical applications (i.e., model accuracy, service speed, and operating cost), unifying them under a single mathematical framework. This comprehensive quantitative representation overcomes the limitations of single-dimensional optimization methods (such as minimizing latency or maximizing accuracy only), and more realistically reflects the overall operational efficiency of edge computing systems. This not only provides a clear and explicit long-term optimization goal for subsequent deep reinforcement learning algorithms but also transforms complex, multi-dimensional decision-making trade-offs into a mathematical problem that can be efficiently solved using advanced algorithms, thus providing a solid foundation for achieving globally optimal system performance.

[0018] Furthermore, the final optimization problem is expressed as follows: Δ∈{1,2,…,10}; where M n (t) represents the set of devices covered by base station n in time slot t. This represents the maximum computing resources of base station n. This represents the maximum computing resources of device m, while This represents the computing resources reserved for other entertainment applications, and τ represents the duration of each time slot. Indicates the device m∈M n (t) The computational task O generated at the beginning of time slot t. mn The maximum acceptable delay of (t).

[0019] The beneficial effects of the aforementioned further scheme are as follows: It breaks through the limitations of traditional fixed-frequency updates, intelligently adjusting the update frequency of the digital twin based on real-time dynamic changes in the network environment: increasing updates to ensure model accuracy when system states change drastically, and reducing updates to conserve valuable bandwidth and energy when the state is stable. Secondly, based on this accurate adaptive update model, this scheme can formulate more efficient task offloading and resource allocation strategies, ensuring that computational tasks are processed with minimal latency and energy consumption. Most importantly, the global optimization perspective based on long time slots ensures that the processes of "adaptive update" and "efficient resource utilization" are not isolated but coordinated and forward-looking, thus avoiding long-term performance losses that may be caused by short-term decisions and maximizing the overall system performance.

[0020] Furthermore, step S5 includes the following steps:

[0021] Initialize the evaluation network Q(s) for deciding the update frequency of the digital twin. Q ,a Q |θ e The Q(s) network of critics used for task unloading and resource allocation decisions P ,a P |θ C ) and actor network μ(s P |θ A Simultaneously, initialize the target networks Q′, Q′, and μ′ corresponding to the aforementioned network; initialize the experience replay buffer Ψ for storing the update frequency of the decision digital twin. Q With the experience replay buffer Ψ used for task unloading and resource allocation decisions P ;

[0022] In each time slot, the current observation state will be... As input, the basic actions of task unloading and resource allocation are output using the actor network, and a preset motion noise is superimposed to increase exploratory power, thereby generating the final action to be executed.

[0023] according to The task is unloaded and executed, earning reward points. and the next time slot state and experience Stored into buffer Ψ P ;

[0024] Experience replay buffer Ψ P The number of experiences reached the preset training priming threshold t>γ P The parameters of the critic network are updated by sampling from the buffer and using the loss function L; at the same time, the actor network is updated using the policy gradient ▽J; and the target networks of the actor and critic are updated using soft updates.

[0025] When the current time slot is an integer multiple of the digital twin update frequency, the evaluation network is used to observe the state. As input, calculate the Q-value of all actions; use the ε-greedy greedy strategy method to select actions.

[0026] Execution And receive the next state s respectively Q (t+1) and reward r Q ;Will Save to buffer Ψ Q ;

[0027] Experience replay buffer Ψ Q The number of experiences reached the preset training priming threshold t>γ Q A small batch of empirical samples is randomly sampled from the network and the parameters of the evaluation network are updated using the loss function L(θ). The target network of the evaluation network is hard-updated every certain number of rounds.

[0028] Through continuous iterative optimization, the trained evaluation network, critic network, and actor network are finally obtained, thereby deriving an optimized digital twin dynamic update strategy and task offloading and resource allocation scheme, completing the digital twin update, task offloading, and resource allocation for digital twin-assisted edge computing systems.

[0029] Furthermore, the expression for the reward value under the digital twin frequency update is as follows: Where, r Q This represents the initial reward value. This indicates task unloading and resource allocation in The reward value of a time slot;

[0030] The beneficial effects of the above-mentioned further scheme are as follows: It adopts a hierarchical hybrid strategy deep reinforcement learning framework, which decouples complex problems into layers and intelligently matches the optimal reinforcement learning paradigm to decision-making tasks of different natures (high-level macro-decision uses a high-sample-efficiency non-policy method to obtain a dynamically updated digital twin policy, while low-level micro-decision uses a high-precision actor-commentator method to obtain task offloading and resource allocation policies), significantly reducing the difficulty of solving the problem. This synergistic design of structure and algorithm fully leverages the advantages of various algorithms, ensuring that the model can converge efficiently and stably, ultimately obtaining a global policy superior to traditional single algorithms, and maximizing the overall system performance in dynamic environments. Attached Figure Description

[0031] Figure 1 This is a flowchart of the method of the present invention.

[0032] Figure 2 A model diagram of an edge computing system assisted by digital twins.

[0033] Figure 3 This is a flowchart of the algorithm of the present invention. Detailed Implementation

[0034] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0035] Example

[0036] like Figure 1 As shown, this invention provides a computational offloading scheme based on dynamic updates of digital twins, the implementation method of which is as follows:

[0037] S1. Obtain the location information of the mobile device, the generated task information, its own energy consumption status, and the available resource status information of the base station under the current time slot. The task information is used to calculate the transmission and calculation time of the task, ensuring that the task completion time is less than the time slot length and the maximum constraint time, which is reflected in the seventh constraint of the optimization problem.

[0038] S2. Based on the real-time status information obtained in step S1, establish a communication model to describe the data transmission process and a task computing model to describe the task processing flow, including a digital twin layer computing model, a local computing model, and an edge computing model; at the same time, construct an energy consumption model for the device based on the computing model.

[0039] S3. Construct an optimization function aimed at improving the overall performance of the system. This function integrates three core performance indicators: the accuracy of the digital twin, the task execution latency, and the energy consumption.

[0040] S4. By defining the multi-objective optimization function described in step S3 for multiple consecutive time slots, and summing the functions of all time slots over a long period of time, the final long-term dynamic optimization problem is constructed. The optimization objective of this problem is to maximize the accuracy of the digital twin and minimize the weighted sum of task execution delay and system energy consumption.

[0041] S5. Based on the final optimization problem, a hierarchical hybrid strategy deep reinforcement learning framework is used to solve the final optimization problem constructed in step S4. This framework combines policy and non-policy learning mechanisms to jointly optimize computational offloading decisions, resource allocation strategies, and the update frequency of the digital twin, ultimately obtaining the optimal task offloading scheme. The implementation method is as follows:

[0042] Initialize the evaluation network Q(s) for deciding the update frequency of the digital twin. Q ,a Q |θ e The Q(s) network of critics used for task unloading and resource allocation decisions P ,a P |θ C ) and actor network μ(s P |θ A Simultaneously, initialize the target networks Q′, Q′, and μ′ corresponding to the aforementioned network; initialize the experience replay buffer Ψ for storing the update frequency of the decision digital twin.Q With the experience replay buffer Ψ used for task unloading and resource allocation decisions P ;

[0043] In each time slot, the current observation state will be... As input, the basic actions of task unloading and resource allocation are output using the actor network, and a preset motion noise is superimposed to increase exploratory power, thereby generating the final action to be executed.

[0044] according to The task is unloaded and executed, earning reward points. and the next time slot state and experience Stored into buffer Ψ P ;

[0045] Experience replay buffer Ψ P The number of experiences reached the preset training priming threshold t>γ P The critic network is trained by sampling from the experience replay buffer and using the loss function L; the policy gradient is then utilized. To update the actor network; to use soft updates to update the target networks of actors and critics;

[0046] When the current time slot is an integer multiple of the digital twin update frequency, the evaluation network Q(s) is used. Q ,a Q |θ e ) Observation status As input, calculate the Q-value of all actions; use the ε-greedy greedy strategy method to select actions.

[0047] Execution And receive the next state s respectively Q (t+1) and reward r Q ;Will Save to buffer;

[0048] The number of experiences in the experience replay buffer reaches the preset training initiation threshold t > γ. Q A small batch of empirical samples is randomly sampled from the network and the parameters of the evaluation network are updated using the loss function L(θ). The target network of the evaluation network is hard-updated every certain number of rounds.

[0049] Through continuous iterative optimization, the trained evaluation network, critic network, and actor network are finally obtained, thereby deriving an optimized digital twin dynamic update strategy and task offloading and resource allocation scheme, completing the digital twin update, task offloading, and resource allocation for digital twin-assisted edge computing systems.

[0050] In this embodiment, as Figure 2 As shown, this system is a novel non-real-time updated digital twin-assisted dynamic edge computing network framework, consisting of a physical world layer and a digital twin layer. In the physical layer, there are N base stations, M mobile devices, and H high-traffic hotspot buildings. The base stations are interconnected via high-speed fiber optic links and communicate with the devices via wireless links. The devices are mobile, and their movement is influenced by the attraction of the hotspot buildings. Each base station is equipped with a resource-constrained edge server to provide computing services to devices within its coverage area. In the digital twin layer, a digital model is constructed to simulate the behavior of physical entities, including devices, hotspot buildings, and base stations.

[0051] Kinematic description of equipment movement:

[0052] Given humans' inherent social and active nature, people's behavior tends to gravitate towards vibrant, high-traffic areas, such as bustling stadiums and lush parks. We modified the traditional APF model based on a gravitational model to describe this tendency of people congregating around high-traffic areas.

[0053] At time slot t, the formula for calculating the attractive force generated by the unvisited hotspot building closest to device m in H is: Where δ represents the attraction ratio factor, and The distance between h and m at time slot t is represented by H, where H represents the set of hotspot buildings.

[0054] Then, the position of device m at the beginning of time slot t+1 can be represented as Where υ represents the moving speed of the device, and τ represents the length of the time slot.

[0055] Communication model:

[0056] Transmission rate is a key indicator for evaluating the link quality between ground equipment and base stations, and its mathematical expression is: Among them, h mn (t) represents the channel gain calculation of the link between device m and base station n in time slot t, R mn (t) represents the communication rate between base station n and a device located within the coverage area of ​​base station n in time slot t, v c A represents the speed of light. d f represents the antenna gain. c For the carrier frequency, d e Indicates path loss index, This indicates that at time slot t, base station n and device m∈M n The Euclidean distance between (t) and B is the wireless channel bandwidth, p mLet m represent the transmit power of device m, N0 represent the Gaussian noise power spectral density, and I represent the intra-cell interference between devices transmitting to the same base station simultaneously.

[0057] Computational model:

[0058] In the first time slot t of each Δ time slot, device m needs to transmit its personal information to the digital twin layer to update its digital model. We use a linear function to map the device's personal information and calculate the amount of digital twin update data that needs to be uploaded in the first time slot t of each Δ time slot. The calculation formula is expressed as follows: Where, α f The data size per unit of computing resource (e.g., 0.5KB / CPU) is quantified, α o This represents the amount of data requested for each task unloading request (e.g., 2KB / request), α ι The location data precision was simulated (e.g., 10 bytes / meter). This represents the available computing resources (O) of device m. mn (t) represents a summary of the computation task, ι m (t)=(x m (t),y m (t) represents the position of device m at the beginning of time slot t.

[0059] The update time of a digital twin, which is the time it takes for the device's personal information to be transmitted to the base station, can be expressed as: t mod Δ = 0. Then, the execution of the task can be simulated in the digital twin layer, thus obtaining the estimated time for local computation and the estimated time for edge computing modes: and Where D mn (t) represents the data size of the task, C mn (t) represents the CPU cycles required to process one bit of data, f mn (t) and f m (t) represents the task O assigned by the digital twin layer guidance base station n and device m in time slot t. mn The computational resources of (t).

[0060] To accurately measure the overall performance of the system, the following was set: This represents the estimated deviation between the resource allocation of the digital twin layer and the actual resource allocation; where β represents a constant of the slope of an adjustment function F(Δ).

[0061] Description of the actual task execution process:

[0062] When the task is offloaded to the base station for computation, the communication delay between the base station n and the mobile device m in time slot t can be expressed as: The actual edge computing latency can be expressed as Where D mn (t) represents the data size of the task, C mn (t) represents the CPU cycles required to process one bit of data; This indicates the estimated deviation between the resource allocation of the digital twin layer and the actual resource allocation.

[0063] When the task is computed locally, the actual local computation latency can be expressed as:

[0064] The total transmission delay between mobile terminal m and base station n in time slot t can be expressed as: Where x mn (t)∈{0,1} represents a binary unloaded variable, x mn (t) = 1 indicates that task O mn (t) is offloaded to base station n for execution in time slot t, while x mn (t) = 0 indicates that task O mn (t) is calculated locally.

[0065] The expression for the communication energy consumption of mobile device m in time slot t is as follows: The expression for calculating the energy consumption of device m in time slot t is as follows: The expression for the total energy consumption of device m is as follows: Where, p m Let κ represent the transmit power of the mobile device m, and let κ represent a constant of the effective CPU switching capacitor.

[0066] Ultimately, the overall performance can be expressed as: Where ω1, ω2, and ω3 represent the weighting coefficients for digital twin bias, delay, and energy consumption, respectively.

[0067] Based on the above digital twin update and task execution process, it can be seen that by adjusting the task offloading decision, base station resource allocation, and digital twin update frequency within each specified time period, a balance can be achieved between controlling task latency, energy consumption, and digital twin accuracy.

[0068] In summary, the optimization objective of this invention is to finely control the update frequency of the digital twin layer to Δ, and the integer unloading vector to be X = {x}. mn The resource allocation strategy is F = {f(t)}. mn (t),f m To maximize overall performance, we need to find the optimal solution for the optimization problem. Therefore, the specific optimization problem is formulated as follows: Among them, M n(t) represents the set of devices covered by base station n in time slot t. This represents the maximum computing resources of base station n. This represents the maximum computing resources of device m, while This represents the computing resources reserved for other entertainment applications, and τ represents the duration of each time slot. Indicates the device m∈M n (t) The computational task O generated at the beginning of time slot t. mn The maximum acceptable delay of (t).

[0069] Due to the NP-hard nature and time-coupled characteristics of this problem, traditional optimization algorithms struggle to handle it. Therefore, this invention utilizes deep reinforcement learning (DRL) algorithms, which can autonomously learn complex policies and manage high-dimensional, nonlinear features to solve complex problems in dynamic environments. Although the training process of the DRL algorithm takes time, the trained model can quickly generate optimized or feasible solutions using existing data, thereby rapidly adapting to real-time or dynamic environments and providing reliable results.

[0070] To efficiently solve the aforementioned NP-hard, time-coupled long-term dynamic optimization problem, this invention proposes a hierarchical hybrid policy deep reinforcement learning scheme. The core of this scheme is to decompose the complex joint optimization problem at both the time and decision levels, and then match the theoretically optimal reinforcement learning paradigm to the characteristics of different sub-problems, thereby achieving efficient, stable, and accurate policy learning. Traditional "flattened" reinforcement learning methods face the "curse of dimensionality" and the "credit allocation problem" when dealing with complex problems involving both long-term strategic planning and immediate tactical execution. This means that the state-action space is too large, leading to low learning efficiency and difficulty in judging the merits of macro-level decisions. This invention overcomes these limitations by decomposing the original problem into two levels: the higher level uses the DQN algorithm to handle macro-level policy formulation, i.e., the optimal update frequency Δ of the decision digital twin. This is a low-frequency, high-impact strategic decision, the effects of which need to be evaluated over a relatively long time window. The lower level uses the DDPG algorithm to handle micro-level action execution, i.e., within each time slot, based on the macro-level policy given by the higher level (in this case, the update cycle), specific task unloading and resource allocation are performed. This is a high-frequency, immediate tactical decision. This hierarchical structure allows higher-level agents to focus on the instantaneous details of lower-level agents, optimizing their macro-level strategies based solely on the cumulative performance feedback from the lower-level agents over a single cycle. Simultaneously, the decision space of the lower-level agents is effectively simplified, as they operate within a fixed macro-level framework. This decoupling significantly reduces learning complexity and stabilizes the training process.

[0071] In this embodiment, the optimization problem proposed by the present invention is, given an environmental state, to optimize the execution of actions to achieve the objective function. maximize.

[0072] In this embodiment, the digital twin update frequency, task unloading, and resource allocation strategy are obtained, such as... Figure 2 As shown, the specifics are as follows:

[0073] 1) Sensing the location information of the mobile device, the task information to be processed, its own energy consumption status, and the available resource status information of the base station in the current time slot;

[0074] 2) Based on the communication model, calculation model, and energy consumption model for time slot t, the digital twin bias, task execution time, and energy consumption time are obtained, and a comprehensive performance index function is constructed.

[0075] 3) Based on each time slot n t and Φ m (t) Construct a long-term dynamic optimization problem;

[0076] 4) Construct the underlying Markov decision process (task unloading decision and resource allocation). The expression for the detailed environmental information observable in time slot t is as follows: Among them ι m (t) represents the location of device m in time slot t. and f n (t) represent the available computing resources for device m and base station n at time slot t, respectively. mn (t) represents task O mn The data size of (t), while E m (t) represents the total energy consumption at time slot t. Furthermore, the action that the underlying agent can perform is defined as a. P (t)={x mn (t),f mn (t),f m (t)}, where x mn (t) represents the binary unloaded variable, f mn (t) and f m r(t) represents the resource allocation strategy for base station n and mobile device m, respectively. Furthermore, r(t) represents the action a performed in time slot t. P The immediate reward obtained after (t). This reward is directly related to the objective function of a single time slot, and its expression is: That is, the negative value of the instantaneous comprehensive cost of the system.

[0077] 5) Construct a high-level Markov decision process (digital twin update frequency). The expression for the detailed environmental information observable in time slot t is as follows: Among them ι m (t) represents the location of device m in time slot t. This represents the energy consumed in the past Δ time slot. Furthermore, the actions that the higher-level agent can perform are defined as a. Q (t) = {Δ} (1 ≤ Δ ≤ 10). Furthermore, R Q Let represent the reward function, where Indicates that during the execution of action a Q After (t), the state changes from s Q (t) transitions to s Q The instant reward value at (t+1).

[0078] 6) The hierarchical hybrid strategy deep reinforcement learning framework (FORO-HDRL) proposed in this invention, such as... Figure 3 As shown, the complete training and execution process is as follows. In the iterative training phase: the goal of this phase is to learn the optimal policy network that maximizes long-term cumulative rewards through continuous interaction between the agent and the environment.

[0079] ● Initialize the evaluation network Q(s) for the high-altitude UFO problem Q ,a Q |θ e ) and its corresponding target network Q′(s Q ,a Q |θ e′ And make the initial parameters the same, i.e., θ e′ ←θ e Initialize the actor network μ(s) for the underlying ORO problem. P |θ A ), critic network Q(s) P ,a P |θ C ) and its corresponding target network μ′(s P |θ A′ ) and Q′(s P ,a P |θ C′ And make the initial parameters the same, i.e., θ A′ ←θ A ,θ C′ ←θ C .

[0080] ● Set the total number of training steps T total Learning rate and mini-batch size η for each network Q ,η P Discount factor ψ, exploration rate ò, and target network soft update coefficient ω; initialize buffer Ψ for storing high-level macro-decision experience. Q Initialize the buffer Ψ used to store the underlying micro-decision experience. P.

[0081] ●Main Loop: Iterative Interaction and Experience Collection

[0082] ●From t=1 to T total In each time slot, the following interactions are performed:

[0083] ●High-level action selection (periodic decision-making): If the current time slot t is the high-level decision point (i.e., t(modΔ) = 0), then the high-level agent selects action a according to the ò-greedy strategy. Q (t): Where ò is the exploration probability, which decays as training progresses to balance exploration and exploitation.

[0084] ● Performing high-level actions within the environment Q (t), using Calculate the immediate reward value. ● Underlying action selection (per-slot decision): The underlying agent selects the action based on the current state s. P (t), a deterministic action is generated through an actor network, and random noise Y is added to facilitate exploration: a P (t)=μ(s P (t)|θ A )+Y;

[0085] ● Perform low-level actions within the environment. P (t), observing the instantaneous reward r P (t) and the underlying state s of the next time slot P (t+1); the underlying empirical tuple (s) P (t),a P (t),r P (t),s P (t+1) is stored in the experience replay buffer Ψ P .

[0086] ●The underlying reward r P (t) Accumulated reward r added to the current high-level decision-making cycle Q When a high-level decision-making cycle ends, the corresponding high-level experience tuple (s) will be... Q ,a Q ,r Q ,s′Q) are stored in the experience replay buffer Ψ Q .

[0087] ●Depth Neural Network Update (DDPG): Experience Replay Buffer Ψ P The number of experiences reached the preset training priming threshold t>γ PThe parameters of the critic network are updated by sampling from the buffer and using the loss function L; simultaneously, the actor network is updated using the policy gradient ▽J; and the target networks θ of both actors and critics are updated using soft updates. A ; Soft update the target network parameters for actors and critics.

[0088] ●High-level network update (DQN): If the buffer Ψ Q If the number of samples within a given range reaches a preset threshold, then η samples are randomly selected from that range. Q Training is performed on 1 sample: For each sample i, the target value y for evaluating the network is calculated. i : Where ψ represents the discount factor, used to control the degree of influence of future rewards on current rewards; by minimizing the loss function L(θ) e The parameters of the evaluation network are updated using a certain method; every fixed number of steps C, the parameters of the current evaluation network are hard-updated to the target network.

[0089] ● After the above iterative process is completed, the output includes the trained evaluation network Q, actor network μ, and critic network Q, which contain the optimal policy information, and is used to solve the optimization problem.

[0090] The solution process is as follows: This stage utilizes the optimal policy model, whose network parameters have been learned and solidified during the iterative training phase, to quickly and deterministically solve the practical problem. The core idea is that in each time slot, the current environmental state information is input into the trained hierarchical reinforcement learning model, which then rapidly outputs the optimal action. By sequentially advancing through the time slots, the entire optimization problem is ultimately solved.

[0091] The input to the solution process includes preset mobile device trajectories, task arrival information, etc. This information serves as known environmental state changes, providing the basic data for constructing the state vector for each subsequent time slot. Before the solution begins, an empty action sequence A is initialized to record the final decision strategy. In each time slot from t=0 to T, the solution process is as follows: First, the system observes the environmental state of the current time slot t, including mobile device location, task information, and real-time system energy consumption, base station available resources, etc., and formats and encapsulates this information into a high-level state vector s. Q (t) and the underlying state vector s P (t).

[0092] Next, the state vector is input into the already trained hierarchical hybrid policy deep reinforcement learning model. For the high-level digital twin update frequency update problem, a nonlinear transformation is performed on the fully connected layer of the evaluation network, ultimately outputting the Q-values ​​corresponding to all possible discrete actions (i.e., different update frequencies Δ) in the current time slot. At this point, the system uses a masking mechanism to filter the Q-values, setting the Q-values ​​of actions that do not meet the current system constraints to negative infinitesimals, thereby ensuring the effectiveness of the decision. Finally, the action with the largest Q-value is selected as the optimal update frequency Δ under the current macroscopic cycle. * : At the same time, the action is stored in action sequence A.

[0093] Furthermore, for the underlying task unloading decision and resource allocation problem, the state of this time slot is fed into the Actor network (μ). This network directly outputs a deterministic, high-dimensional continuous action vector, which represents the optimal task unloading and resource allocation scheme for the current state. The system executes the calculated optimal low-level action a in the environment. P* (t), and store the action in action sequence A. Subsequently, the environment state transitions to the next time slot t+1, and the above process of state observation, model inference, and action execution is repeated. After all T time slots have been executed, the final output complete action sequence A is the dynamic, globally aware digital twin update, task unloading, and resource allocation scheme obtained by this invention for the entire optimization problem.

[0094] In summary, compared with the prior art, the present invention has the following advantages:

[0095] By constructing digital twin accuracy, task latency, and system energy consumption into a long-term cumulative comprehensive optimization objective and using it as the final reward source for high-level and low-level reinforcement learning agents, this invention enables decision-making from a global time dimension. Specifically, this method overcomes the short-sighted decision-making problem caused by traditional schemes that only focus on instantaneous performance or a single metric (such as minimizing latency). It can dynamically weigh the complex relationships between various performance metrics, ensuring that the final generated strategy is globally optimal or near-optimal throughout the entire task cycle, thereby achieving long-term stability and balance of the overall system performance.

[0096] The core of this invention is a hierarchical hybrid strategy deep reinforcement learning (FORO-HDRL) framework. This framework decomposes the original problem into a high-level digital twin update frequency optimization problem and a low-level task offloading and resource optimization problem. More importantly, this invention matches the most suitable algorithmic paradigm for sub-problems of different natures: at the high level, for the digital twin update frequency optimization problem with a discrete action space, a value iteration method (DQN) centered on the evaluation network Q is adopted; at the low level, for the task offloading and resource optimization problem with a continuous and high-dimensional action space, the actor-critic (DDPG) method is adopted. This hybrid architecture design achieves effective decoupling of complex problems and ensures that each sub-problem can be solved with the most efficient and accurate results, which is difficult for a single algorithm to achieve.

[0097] Furthermore, the training mechanism of this invention is also targeted: a soft-update mechanism is used in the low-level DDPG part to smooth the update of the target network, ensuring the stability of continuous control task learning; while a hard-update mechanism is used in the high-level DQN part. This refined training mechanism design ensures that the entire hierarchical framework can converge stably and efficiently.

[0098] The method proposed in this invention has strong practical value. After offline training, the resulting policy network model exhibits low computational complexity during the actual execution (solution) phase, and can quickly generate optimal dynamic update and task processing strategies based on real-time input environmental states. This "sufficient offline training, rapid online inference" model makes it well-suited for various complex and dynamic scenarios requiring rapid response, such as smart cities and connected vehicles, and possesses high potential for practical deployment.

Claims

1. A computational offloading scheme based on dynamic updates of digital twins, characterized in that, Includes the following steps: S1. Obtain the location information of the mobile device in the current time slot, the task information generated by the device, the energy consumption status of the device, and the available resource status information of the base station; S2. Based on the real-time status information obtained in step S1, establish a communication model to describe the data transmission process and a task computing model to describe the task processing flow, including a digital twin layer computing model, a local computing model, and an edge computing model; at the same time, construct an energy consumption model for the device based on the computing model. S3. Construct an optimization function aimed at improving the overall performance of the system. This function integrates three core performance indicators: the accuracy of the digital twin, the task execution latency, and the energy consumption. S4. By defining the multi-objective optimization function described in step S3 for multiple consecutive time slots, and summing the functions of all time slots over a long period of time, the final long-term dynamic optimization problem is constructed. The optimization objective of this problem is to maximize the accuracy of the digital twin and minimize the weighted sum of task execution delay and system energy consumption. S5. Based on the final optimization problem, a hierarchical hybrid strategy deep reinforcement learning framework is used to solve the final optimization problem constructed in step S4. The framework combines policy and non-policy learning mechanisms to jointly optimize computational offloading decisions, resource allocation strategies, and the update frequency of digital twins, ultimately obtaining the optimal task offloading scheme.

2. The computational offloading scheme based on dynamic updates of digital twins according to claim 1, characterized in that, The expression for the communication is as follows: Among them, h mn (t) represents the channel gain between device m and base station n at time slot t, R mn (t) represents the communication rate between base station n and device m located within the coverage area of ​​base station n in time slot t, v c A represents the speed of light. d f represents the antenna gain. c For the carrier frequency, d e Indicates path loss index, This indicates that at time slot t, base station n and device m∈M n The Euclidean distance between (t) and B is the wireless channel bandwidth, p m Let m represent the transmit power of device m, N0 represent the Gaussian noise power spectral density, and I represent the intra-cell interference between devices transmitting to the same base station simultaneously.

3. The computational offloading scheme based on dynamic updates of digital twins according to claim 1, characterized in that, The expression for the task computation model is as follows: in, This represents the total amount of update data that the device needs to upload in the first time slot t of each Δ time slot. This represents the personal information upload delay between base station n and device m in the first time slot t of each Δ time slot. This indicates the edge computing latency within the digital twin layer. α represents the local computation latency within the digital twin layer. f The data size per unit of computing resource (e.g., 0.5KB / CPU) is quantified, α o This represents the amount of data requested for each task unloading request (e.g., 2KB / request), α ι The location data precision was simulated (e.g., 10 bytes / meter). This represents the available computing resources (O) of device m. mn (t) represents a summary of the computation task, ι m (t) represents the position of device m at the beginning of time slot t, Δ represents the number of time slots in the digital twin update cycle, and D mn (t) represents the data size of the task, C mn (t) represents the CPU cycles required to process one bit of data in the task, f mn (t) and f m (t) represents the task O assigned by the digital twin layer guidance base station n and device m in time slot t. mn The computational resources of (t). in, This represents the communication delay between base station n and mobile device m in time slot t. Indicates the actual edge computing latency. Indicates the actual local computation latency. This represents the communication energy consumption of mobile device m in time slot t. This represents the calculated energy consumption of device m in time slot t. This represents the estimated deviation between the resource allocation of the digital twin layer and the actual resource allocation, where β represents a constant representing the slope of an adjustment function F(Δ). It is an indicator function that indicates when the condition x is true. otherwise p m Let κ represent the transmit power of the mobile device m, and let κ represent a constant of the effective CPU switching capacitor.

4. The computational offloading scheme based on dynamic updates of digital twins according to claim 1, characterized in that, The expression for the comprehensive performance index of the equipment is as follows: in, T represents the transmission time of the task. mn (t) represents the total execution time of the task, E m (t) represents the total energy consumption of device m, Φ m (t) represents the overall performance index. Indicates the bias of a digital twin, describing its accuracy; x mn (t)∈{0,1} represents a binary unloaded variable, x mn (t) = 1 indicates that task O mn (t) is offloaded to base station n for execution in time slot t, while x mn (t) = 0 indicates that task O mn (t) is computed locally.

5. The computational offloading scheme based on dynamic updates of digital twins according to claim 1, characterized in that, The final optimization problem is expressed as follows: Among them, M n (t) represents the set of devices covered by base station n in time slot t. This represents the maximum computing resources of base station n. This represents the maximum computing resources of device m, while This represents the computing resources reserved for other entertainment applications, and τ represents the duration of each time slot. Indicates the device m∈M n (t) The computational task O generated at the beginning of time slot t. mn The maximum acceptable delay of (t).

6. The computational offloading scheme based on dynamic updates of digital twins according to claim 1, characterized in that, S5 includes the following steps: Initialize the evaluation network Q(s) for deciding the update frequency of the digital twin. Q ,a Q |θ e The Q(s) network of critics used for task unloading and resource allocation decisions P ,a P |θ C ) and actor network μ(s P |θ A Simultaneously, initialize the target networks Q′, Q′, and μ′ corresponding to the aforementioned network; initialize the experience replay buffer Ψ for storing the update frequency of the decision digital twin. Q With the experience replay buffer Ψ used for task unloading and resource allocation decisions P ; In each time slot, the current observation state will be... As input, the basic actions of task unloading and resource allocation are output using the actor network, while a preset motion noise is superimposed to increase exploratoryness, thereby generating the final action to be executed. according to The task is unloaded and executed, earning reward points. and the next time slot state and experience Stored into buffer Ψ P ; Experience replay buffer Ψ P The number of experiences reached the preset training priming threshold t>γ P The parameters of the critic network are updated by sampling from the buffer and using the loss function L; the actor network is updated using the policy gradient; and the target networks of the actor and critic are updated using soft updates. When the current time slot is an integer multiple of the digital twin update frequency, the evaluation network is used to analyze the observed state. As input, calculate the Q-value of all actions; use the ε-greedy greedy strategy method to select actions. Execution And receive the next state s respectively Q (t+1) and reward r Q ;Will Save to buffer; The number of experiences in the experience replay buffer reaches the preset training initiation threshold t > γ. Q A small batch of empirical samples is randomly sampled from the network and the parameters of the evaluation network are updated using the loss function L(θ). The target network of the evaluation network is hard-updated every certain number of rounds. Through continuous iterative optimization, the trained evaluation network, critic network, and actor network are finally obtained, thereby deriving an optimized digital twin dynamic update strategy and task offloading and resource allocation scheme, completing the digital twin update, task offloading, and resource allocation for digital twin-assisted edge computing systems.

Images