Dynamic resource configuration method for edge computing system

Abstract

The application discloses a dynamic resource configuration method for an edge computing system, which comprises the following steps: constructing an energy consumption model and a task queue model of a WPT supporting MEC system to generate an optimization problem of joint resource configuration and task scheduling; simplifying the optimization problem by establishing a Markov decision process (MDP); and solving and generating a resource allocation convex optimization scheme by a two-stage deep learning Q network (TS-DQN). The application minimizes the long-term average energy consumption of the system by jointly optimizing the parameters of the system, can effectively solve the long-term energy consumption minimization problem of the system without providing prior information such as channel state distribution and task arrival rate, and ensures successful execution of the task.

Description

Technical Field

[0001] This invention relates to a technology in the field of distributed computing, specifically a dynamic resource allocation method for edge computing systems. Background Technology

[0002] With the evolution and development of mobile communication technology, mobile edge computing (MEC) has the advantage of being closer to users compared to traditional cloud computing by deploying computing, communication and caching resources in a fully distributed manner.

[0003] The main shortcomings and deficiencies of existing MEC systems supporting Wireless Power Transfer (WPT) are as follows: WPT scenarios require MEC systems to incorporate energy control and power supply time allocation, which makes the time-coupled characteristics more complex. This makes it difficult for traditional solutions to obtain long-term optimization strategies, and they can only provide static optimization schemes. These algorithms cannot fully consider the impact of time-varying factors of energy transfer in dynamic wireless environments on MEC systems. Although dynamic optimization schemes based entirely on reinforcement learning consider long-term system benefits, they struggle to cope with the state dimension explosion problem caused by an increase in the number of users accessing the system. Their convergence speed is low in complex problems, and there is room for improvement in solution accuracy. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies that fail to consider the impact of dynamic task queues and rechargeable energy storage devices on WPT allocation, and thus cannot perceive potential long-term patterns in dynamic wireless environments. It proposes a dynamic resource allocation method for edge computing systems, considering MEC scenarios supporting multiple IoT devices powered by WPT, and minimizing the system's long-term average energy consumption through joint optimization of system parameters. Compared to methods relying solely on reinforcement learning, the precise solution of convex optimization leads to better decision-making, and the introduction of action generation strategies further accelerates neural network training, improving the efficiency of model exploration. This invention can effectively solve the proposed long-term energy consumption minimization problem without requiring prior information such as channel state distribution and task arrival rate, while ensuring successful task execution.

[0005] This invention is achieved through the following technical solution:

[0006] This invention relates to a dynamic resource allocation method for edge computing systems. It generates an optimization problem of joint resource allocation and task scheduling by constructing an energy consumption model and a task queue model of an MEC system that supports WPT. Then, it simplifies the optimization problem by establishing a Markov Decision Process (MDP) and solves and generates a convex optimization scheme for resource allocation through a two-stage deep learning Q-network (TS-DQN).

[0007] The TS-DQN includes: a DQN (Deep Q Network) unit, an action generation module, and a convex optimization module. The DQN unit has a target network and a main network with the same structure, and trains the model through a cooperative mechanism during network learning. The action generation module expands an action vector output by the main network into multiple related action vectors and stores the action vector with the highest reward in the experience replay pool. The convex optimization module, based on partial decisions output by the main network, transforms the original non-convex problem into a time-independent problem that satisfies convex properties within its scope, and accurately obtains the remaining decisions using a convexity-solving toolkit.

[0008] The main network is a fully connected multilayer neural network. This multilayer neural network receives state information in the real-time environment and outputs the action with the highest score to provide part of the decision for the MEC system. During training, it randomly samples the historical data accumulated in the experience replay pool and calculates the loss function in combination with the output of the target network. Finally, it uses the stochastic gradient descent method to backpropagate itself to realize the entire learning process.

[0009] The target network has the same fully connected structure as the main network, that is, it serves as a historical reference model for the main network. The target network periodically replaces its own weights with the weights of the main network and outputs the Q-values ​​of relevant actions during training to assist the main network in evaluating learning errors.

[0010] The convex optimization module is designed for solving the optimal value of convex functions based on convex sets. It utilizes the property that a local optimal solution in a convex problem is necessarily a global optimal solution, and uses the second derivative or interior point method from the convexity solution toolkit for optimization to accurately solve the input convex problem.

[0011] The method specifically includes the following steps:

[0012] i) Initialize the system environment, generate the initial state S0, establish an empty experience replay pool, and initialize the weights θ and θ′ of the main network and target network of TS-DQN.

[0013] ii) At the start of each time slot, update the task queue R based on the amount of newly arrived tasks. rem，t，jThe channel gain G is updated based on the probability model and the preset state interval. t .

[0014] iii) Change state S t Input is fed into the main network of TS-DQN, and output action A. t .

[0015] iv) Extend the original action A using action generation strategies t .

[0016] v) Input the generated actions sequentially into the convex optimization module of TS-DQN, calculate the total energy consumption of this execution, and retain the action A′ with the highest reward through the reward function. t .

[0017] vi) The system based on action A′ t The execution results update the task queue and energy queue, and save the experience (S). t A′ t R t S t+1 ) to the experience replay pool.

[0018] vii) The target network of TS-DQN is randomly sampled from the experience pool, and its loss function is calculated relative to that of the main network to train the main network. The main network periodically updates the target network weights θ′ using its own weights θ.

[0019] viii) Repeat steps ii) to vii) until the specified number of iterations is completed.

[0020] Technical effect

[0021] This invention combines the deep reinforcement learning method DQN with convex optimization, allowing DQN to learn channel allocation and task scheduling strategies. The original problem is transformed into a time-independent convex problem, and the remaining decisions are calculated through a convex optimization module. Finally, the DQN is evaluated and trained based on execution energy consumption feedback, enabling the solution to operate long-term. Simultaneously, by expanding the multi-ary action vector output by the reinforcement learning model into multiple vectors and storing the action with the highest reward in an experience pool for DQN training, the diversity of actions is increased, accelerating the convergence speed of reinforcement learning. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention;

[0023] Figure 2 This is a scenario diagram of a multi-device MEC system supporting WPT power supply, as shown in the example.

[0024] Figure 3 This is a schematic diagram of the TS-DQN algorithm flow for an example.

[0025] Figure 4 This diagram illustrates how the average energy demand at the base station varies with the total available bandwidth of the system.

[0026] Figure 5 This diagram illustrates how the CPU operating frequency of an edge server changes with the average number of tasks arriving. Detailed Implementation

[0027] like Figure 1 As shown, this embodiment relates to a dynamic resource allocation method for edge computing systems. Figure 5 In the MEC system supporting WPT shown, the grid-powered multi-antenna base station consists of J single-antenna IoT devices. To provide services, the base station is equipped with an edge server to handle tasks that would otherwise be offloaded from the computing devices.

[0028] Each IoT device has a queue for storing task data and a rechargeable battery.

[0029] Step 1) System time slot initialization: The MEC system divides the infinitely long time domain based on discrete time slots. Specifically, each time slot is indexed by t, and its duration is τ. The total system bandwidth W is uniformly channelized into K sub-channels. Collect the channel gain of all sub-channels in the system, where: g t，j，k It is the channel gain of the k-th uplink sub-channel responsible for transmitting data from the j-th device to the base station in the t-th time slot; an IoT device can offload task data in parallel through multiple sub-channels, but each sub-channel can only be allocated to one device in the same time slot.

[0030] At the start of each time slot, each IoT device receives a certain size of computationally intensive task and caches the task data locally. During the duration of each time slot, these cached task data are executed on the local device or remotely on the edge server according to the first-in-first-out (FIFO) principle.

[0031] In Phase I, all IoT devices draw power from the base station to power either the local execution or offloading of the task. Considering the integrity of the task and the correlation between data, the task must maintain its overall form and cannot be arbitrarily fragmented when choosing between local and remote execution. In the case of task offloading, the offloading process is scheduled for Phase II of the time slot, and then the task is executed on the edge server in Phase III. When the task is processed locally, the processing takes place in both Phase II and Phase III. Since the results of task execution are typically small in volume, the time and energy spent sending and receiving results are negligible.

[0032] Step 2) Construct a model for task computation time and system energy consumption, specifically:

[0033] 2.1) During the first phase of time slot t, the energy transmitted from the base station to device j Where: φτ is the duration of stage I, i.e., WPT, and the time segmentation ratio φ∈[0,1], P e，t，j It is the radio frequency energy transmitted to device j, and satisfies ∑ j P e，t，j ≤P max P max The specified base station is used for the maximum transmit power of WPT, η∈(0,1) is the energy conversion efficiency of the device, H t，j This is the channel gain from the base station to device j. During the remaining time (1-φ)τ, the device either uses the uplink wireless channel to offload the task to the edge server, or chooses not to occupy any channel and execute the task locally.

[0034] 2.2) Local task computation time of computing device j Where: R pro，t，j (Unit: NAT) represents the size of the task that device j needs to process within time slot t, β represents the number of device CPU cycles required to compute a unit of NAT data in the task, and a t，j，k ∈{0,1} is a channel allocation indicator. When its value is equal to 1, the system decides to allocate the k-th subchannel to the j-th device for task offloading in the t-th time slot; otherwise, its value is set to 0. The CPU frequency of device j in time slot t is f. u，t，j Its size can be flexibly adjusted using Dynamic Voltage and Frequency Scaling (DVFS); u，max The maximum CPU frequency that the device can achieve, satisfying constraint f u，t，j ≤f u，max Indicator functions When condition a t，j，1 +a t，j，2 +...+a t，j，K ≥1 is satisfied, otherwise To maintain task integrity while allowing device j to unload its task, the local processing time is T. loc，t，j =0.

[0035] 2.3) Calculate the total energy consumed by device j in time slot t, that is, the energy collected by the device in phase I for local computation and / or offloading operations of the task, specifically: the total energy consumed by device j in time slot t. Among them: local computing energy consumption k jLet the effective switched capacitor coefficient of the j-th device be denoted as , then the locally calculated power of device j in time slot t is denoted as . When device j selects to offload its task and executes it remotely on the edge server, the transmit power used for offloading transmission in the t-th time slot... T tr，t，j For transmission time, σ 2 It is the noise power variance of the base station. This is the total bandwidth allocated to device j; the corresponding energy consumption used for offloading is then obtained.

[0036] When f c，t，j It is the server CPU frequency allocated for executing the offloading task of device j, and satisfies ∑ j f c，t，j ≤f c，max f c，max The server task computation time for device j is the maximum CPU processing capacity of the server.

[0037] 2.4) Calculate the energy consumption of the server when remotely executing tasks.

[0038] in: δ is the power of device j running task on the edge server, and δ is the effective switching capacitor coefficient of the server.

[0039] Step 3) Construct the task queue model: Based on R t，j Let R be the number of new tasks arriving at device j in time slot t, and the device will cache these arriving tasks. During the duration of the time slot, a portion of the cached tasks are sent to the edge server, or performed locally on the device according to the FIFO principle. rem，t，j ≤R max Let R be the length of the task queue for device j in time slot t, where: max This refers to the capacity of the buffer; the buffer queue receives newly arriving tasks in each time slot and outputs a portion of the stored tasks for execution, i.e.: Where: due to the causal relationship, the inequality R is satisfied. pro，t，j ≤R rem，t，j +R t，j .

[0040] Use E bat，t，j ≤E bmax The state-of-charge (SoC) of device j at the start of time slot t is defined as follows: E bmaxThe battery capacity of the device. In each time slot t, the battery of device j is first charged by WPT, and then power is consumed through local calculation or offload calculation. The update process is as follows:

[0041] When all batteries are fully charged during the initial system operation, that is... Furthermore, the total energy consumption of each IoT device cannot exceed the available energy it stores in its battery or captures using WPT, i.e.

[0042] Step 4) Constructing the optimization problem of joint resource allocation and task scheduling: Minimize the time-averaged energy demand of the WPT-MEC system in an infinite time domain, while ensuring the causal relationship between energy and tasks for each device, i.e., satisfying: 1) The energy consumption of a device when offloading and executing tasks locally cannot exceed the effective value; 2) The size of the tasks planned to be processed in each time slot cannot exceed the size of the currently pending tasks, and these tasks can be successfully processed within the duration τ of the time slot. Thus, the optimization problem is:

[0043]

[0044] in: N is the number of time slots, and the base station's transmit power used for WPT. Task size to be processed Channel allocation flags used for offloading tasks Uninstallation duration Base station CPU operating frequency and the CPU operating frequency of IoT devices

[0045] The above constraints respectively describe the dynamic SoC of the battery, guarantee the causal relationship of energy, describe the dynamic evolution process of the task data queue, guarantee the causal relationship of task data, describe the delay constraints, ensure that the planned task data can be completed within any time slot, and limit the maximum allocable amount of the corresponding CPU operating frequency and transmission power.

[0046] Step 5) The optimization problem in Step 4 is simplified by establishing a Markov Decision Process (MDP). Specifically, the solution process is regarded as an MDP, and the channel allocation decision A is determined by TS-DQN. c，t and task scheduling decision R pro，tAfter obtaining these two decisions, the remaining part of the optimization problem is transformed into convexity and can be solved using readily available convex problem optimization toolkits (such as interior point methods).

[0047] The MDP is built in an infinite time domain and consists of a tuple with four elements (S, A, P). r R e ), where: the system's state element S can be observed at the beginning of each time slot t, and is used Distinguish, among which: Record the arrival status of tasks on the device. Record all channel gain states from the device to the base station. Record the battery SoC of each device. For the state space, use This is a concatenation and merging of state vectors. E bat，t The size depends on WPT, offloading scheduling, and resource allocation strategies, while R t With G t It will not be affected by the strategy; for the system's action element A, it is specifically manifested in each time slot t as follows in: For the action space, This represents the binary channel allocation strategy. The size of the task data to be processed, and It is a finite action space. In the proposed TS-DQN framework, actions are obtained from the DQN output; for the state transition probability element P... r This determines the probability of transitions between adjacent time slot states. Reinforcement learning determines action policies based on different system states, π = {π c , π r}, where: π c and π r The channel allocation strategy and task scheduling strategy are described respectively. Since the transmission power required for unloading can directly determine the energy storage status of the device in each state, where: P r {E bat，t+1，j |E bat，t，j , π}=1; For each time slot, the reinforcement learning agent obtains the reward element R based on the current state and the action performed. e It directly reflects the benefit gained from the current action. To minimize the time-averaged energy requirement in the system, the reward function should be negatively correlated with the objective function. In time slot t, the instantaneous reward is defined as the instantaneous objective function taking a negative value, specifically: The action {A} is obtained through DQN. c，t R pro，tAfter that, the immediate reward for each time slot will be used to obtain the remaining optimization variable {P} based on the convex optimization process. e，t T tr，t f c，t f u，t The results were obtained through post-evaluation.

[0048] The TS-DQN described above uses the action value function Q in each time slot t. π (S t A t This is used to calculate the expected cumulative discount reward over an infinite time domain. Where: γ∈[0,1] is the discount factor, π * =argmax A Q(S t A t Describe the optimal strategy.

[0049] The TS-DQN described above performs an update at the beginning of each time slot, that is, the reinforcement learning agent observes the network state S in time slot t. t Action A t Instant Rewards R e，t and the result state S in the next time slot t+1 t+1 Update the Q function in real time, that is: Where: α t ∈[0,1) is the learning rate, which can be adjusted according to the learning performance of reinforcement learning.

[0050] Preferably, in this embodiment, TS-DQN, as a model-free DRL scheme, uses a deep neural network (DNN) to estimate Q-values ​​instead of directly using a Q-table to record all Q-values. This not only solves the problem of adapting to a huge state space but also enhances the model's generalization ability. In each time slot, DQN obtains the optimal action based on the input state by maximizing the Q-value. The estimated Q-value is Q(S, A) ≈ Q(S, A; θ), where: θ is the set of model weights in DQN.

[0051] The TS-DQN includes a target network and a main network, and adopts a dual-network working mode with the same structure, wherein: the target network generates the target Q value that directly affects the selection of actions: Wherein: θ′ records the weight parameters in the target network; these weight parameters will be covered by the weights of the main network over a relatively long time period (e.g., every 10 time slots), i.e., θ′=θ.

[0052] The main network aims to minimize the loss function and uses stochastic gradient descent to update its weight parameters. Its loss function is...

[0053] The TS-DQN described in this embodiment is trained in the following way: based on the experience learned through iterative storage and accumulation in the experience replay pool (S t A t R t S t+1 The model is trained by randomly sampling samples to break the correlation between samples, allowing it to better explore different states in the environment and converge better during training.

[0054] Preferably, the TS-DQN introduces an action generation strategy to optimize training, specifically: expanding an action vector into multiple action vectors with certain correlations, and storing the action vector that receives the highest reward in the experience replay pool for use in subsequent DQN training. This allows for the evaluation of more candidate actions in each network iteration, achieving action diversification and accelerating network convergence. Specifically, this includes:

[0055] i) For the task execution action vector, when the initial action before action expansion is Set the final total number of action vectors to be obtained to the same value. The dimension J is equal. To generate the remaining (J-1) related actions, a finite action space is first defined. The expected value, i.e. After that, The elements in the vector are reordered to obtain the sorted vector. in: And it meets the sorting criteria:

[0056] ii) For Each element in the newly generated action vector should be specified according to the following rules:

[0057] in: and They are respectively In action space The next smaller element and the larger element are labeled with d = 1, 2, ..., J-1 for (J-1) newly generated action vectors.

[0058] iii) For a binary action A with action space {0, 1} and cardinality JK c，t Alternatively, the proposed action generation algorithm can be used to generate the remaining (JK-1) action vectors, ultimately yielding a total of (JK+J) action vectors related to A. c，t With Rpro， The number of possible actions obtained after merging the two types of actions is as high as J × JK = J 2 K.

[0059] To reduce computational complexity, the implementation first randomly samples (JK+J) actions from the generated action vector, then inputs the sampled actions into a reward evaluation module based on convex optimization, and finally saves the action corresponding to the highest reward to the experience replay pool for use in subsequent model training.

[0060] Step 6) Generate a convex optimization scheme for resource allocation based on the trained TS-DQN, specifically: In time slot t, given the decision {A} obtained from the trained TS-DQN... c，t R pro，t After that, a simplified convex problem of the optimization problem is generated. The optimal solution of the remaining decisions is quickly obtained by using the existing convex problem solving toolkit, and a global optimized resource allocation scheme is obtained.

[0061] The simplified convex optimization model is as follows: stE loc，tj +E tr，t，j ≤E bat，t，j +E wpt，t，j T tr，t，j +T edg，t，j ≤(1-φ)τ,,T loc，t，j ≤(1-φ)τ,,f u，t，j ≤f u，max ,,∑ j f c，t，j ≤f c，max ,,∑ j P e，t，j ≤P max , where: E loc，t，j E edg，t，j T loc，t，j和 T edg，t，j Regarding f c，t and f u，t For joint convexity, the offload transmission power of device j using subchannel k in the t-th time slot is: The offloading transmission power is about The convex function, and its projection function, i.e., the unloading energy consumption. Regarding T tr，t，j In ∑ k r t，j，k ≤R pro，t，j The range is also convex. Therefore, the entire problem is a convex problem. The remaining decision variables {P} e，t T tr，t f c，t f u，tIt can be solved efficiently using the standard convex solver toolkit. The reward of TS-DQN is taken from the negative of the minimum of the objective function.

[0062] Preferably, when the collected energy is insufficient to support the current task scheduling strategy R pro，t Or because the backlog of task data exceeds the size of the limited buffer R. max This can lead to task buffer overflow and data loss. Task execution failure requires retransmission and recalculation, incurring additional costs. This embodiment addresses task failure in the reward function. Let F... t，j Indicates whether device j failed while performing a task in time slot t, when F t，j =1, then the task execution failed; otherwise, F t，j =0. Simultaneously, a penalty term F is added to the reward function. t，j C f To adjust it, that is: R e，t ←R e，t -F t，j C f , where: C f The penalty amount for failing a task.

[0063] Through specific practical experiments, the performance of the proposed TS-DQN scheme was evaluated using a series of simulation experiments. Specifically, this included configuring relevant parameters in a WPT-MEC system model and then evaluating the proposed TS-DQN scheme based on different parameter adjustments. The system's total channel bandwidth W = 2MHz was considered, including 3 IoT devices and 3 sub-channels. The experiment set the highest CPU frequencies of the MEC server and IoT devices to f0 and f1 respectively. c，max =4GHz and f u，max =1GHz, with each time slot lasting τ=0.2s. In addition, the duration ratio of the WPT phase is set to φ=0.4, the WPT energy transfer attenuation factor η=0.3, the number of CPU cycles required to process each NAT unit of data is β=1000, and k=δ=10. -26 , σ 2 =10 -8 WPT maximum transmission power P max =6W, the additional penalty coefficient C for DQN gains when task execution fails. f =50. For each time slot, the task arrival rate is taken from a continuous interval. (Unit: Knats), channel gain is taken from... (Unit: dB) The battery can store a maximum of E energy. bmax =16mJ, the amount of tasks that each time slot can handle is taken from a discrete set. The maximum length of the task data cache queue is R. max = 6.4 Knats. The state selection probability follows a uniform distribution for task arrival rate and wireless channel gain. The proposed TS-DQN contains two hidden layers, each with 256 neurons. The playback memory pool size D = 2000, the learning rate of the neural network α = 0.001, and the discount factor γ for Q-value updates γ = 0.9.

[0064] To further demonstrate the performance of the scheme, this embodiment compares TS-DQN with three baseline experiments: "local computation only," "full offload," and "DDPG." For the "local computation only" scheme, each time slot only considers executing incoming tasks locally. For the "full offload" scheme, IoT devices evenly distribute system bandwidth and offload all tasks to servers for edge computing. The "DDPG" scheme uses the DDPG (Deep Deterministic Policy Gradient) method to optimize all decisions. DDPG is one of the latest DRL models, combining the network structures of Actor-Critic and DQN, and can support continuous optimization of decision variables, thus addressing the challenges of large action spaces.

[0065] like Figure 4 The figure shows the average energy consumption of the WPT-MEC system under the TS-DQN method and three other comparative schemes (local computation only, full offloading, and DDPG-based scheme) as the total available bandwidth of the system increases. Analysis of the figure shows that TS-DQN consistently maintains the lowest energy consumption under different parameters, thus outperforming the other three benchmark schemes. Further analysis reveals that the energy consumption of TS-DQN, when the available bandwidth is large, is only 44.1%, 64.5%, and 88.2% of that of local computation only, full offloading, and DDPG, respectively. Unlike the local computation only and full offloading schemes, the proposed TS-DQN and DDPG-based schemes determine task scheduling by learning channel state, task arrival status, and battery energy storage information, which significantly saves system energy consumption. On the other hand, the DDPG-based scheme relies solely on DRL to obtain an approximate optimal solution, while the proposed TS-DQN can accurately derive the optimal decisions for WPT transmission power, CPU frequency, and offloading time by utilizing the convexity of the decomposed problem, thus surpassing DDPG in performance.

[0066] In terms of overall performance, except for local computing-only schemes, the average energy consumption of task offloading decreases as the total available bandwidth of the system increases. For example... Figure 5As shown, the average energy demand for different schemes increases with the average number of tasks arriving. It can be observed that all curves rise with increasing task arrivals, as the increased task load leads to higher computational and transmission overhead. Compared to local computation only and full offloading schemes, the proposed TS-DQN shows increasingly significant performance gains with increasing task arrivals. When the task arrival volume is large, TS-DQN reduces average energy consumption by nearly 60% compared to the full offloading scheme. Furthermore, the proposed TS-DQN incorporates convex optimization into DQN, demonstrating its superior energy reduction compared to DDPG-based schemes.

[0067] In summary, this invention proposes a novel WPT-MEC optimization scenario that requires overall coordination of the base station's WPT power, IoT device offloading plans, wireless channel allocation, and CPU configuration of edge servers and local devices to minimize the system's long-term energy consumption. This invention addresses the dynamic nature of the environment by considering time-varying wireless channel gain, dynamic task arrival, and battery devices capable of storing energy. A TS-DQN algorithm is designed to effectively solve the problem, enabling the solution of the proposed long-term energy consumption minimization problem without requiring prior information such as channel state distribution and task arrival rate, while ensuring successful task execution.

[0068] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A dynamic resource allocation method for edge computing systems, characterized in that, By constructing an energy consumption model and a task queue model for an MEC system that supports WPT, an optimization problem of joint resource allocation and task scheduling is generated. Then, the optimization problem is simplified by establishing a Markov decision process, and the resource allocation convex optimization scheme is generated by solving it through TS-DQN. The TS-DQN includes: a DQN unit, an action generation module, and a convex optimization module. The DQN unit has a target network and a main network with the same structure, and trains the model through a cooperative mechanism during network learning. The action generation module expands an action vector output by the main network into multiple related action vectors and stores the action vector with the highest reward in the experience replay pool. The convex optimization module, based on partial decisions output by the main network, transforms the original non-convex problem into a time-independent problem that satisfies convex properties within its scope, and accurately obtains the remaining decisions using a convexity-solving toolkit. The simplification refers to simplifying the optimization problem by establishing a Markov decision process, specifically by treating the solution process as an MDP and using TS-DQN to determine the channel allocation decision. and task scheduling decisions After obtaining these two decisions, the remaining part of the optimization problem is transformed into convexity, and can be solved using readily available convex problem optimization toolkits. Then, based on the trained TS-DQN, a convex optimization scheme for resource allocation is generated, specifically: in time slots... Given the decision obtained by the trained TS-DQN Then, a simplified convex problem of the optimization problem is generated, and the optimal solution of the remaining decisions is quickly obtained using the existing convex problem solving toolkit, thus obtaining the global optimized resource allocation scheme; The TS-DQN introduces an action generation strategy to optimize training. Specifically, it expands an action vector into multiple related action vectors and stores the action vector with the highest reward in an experience replay pool for use in subsequent DQN training. This allows for the evaluation of more candidate actions in each network iteration, achieving action diversification and accelerating network convergence. Specifically, this includes: i) For the task execution action vector, when the initial action before action expansion is Set the final total number of action vectors to be obtained as the same as... Dimensions Equal; to generate the rest For each related action, first define a finite action space. The expected value, i.e. After that, regarding The elements in the vector are reordered to obtain the sorted vector. And it satisfies the sorting criteria: ; ii) For Specify each element in the newly generated action vector according to the following rules: ,in: and They are respectively In action space The next smaller element in the larger element is used with Label A newly generated action vector; iii) For action spaces of And the base number is binary action The proposed action generation algorithm can also be used to generate the remaining actions. Each action vector can ultimately be used to obtain a total of action vectors. A about and The number of possible actions obtained after merging the two types of actions is as high as [number missing]. indivual; The dynamic resource allocation for edge computing systems specifically includes: i) Initialize the system environment and generate the initial state. Establish an empty experience replay pool and initialize the weights of the TS-DQN main network and target network. ; ii) At the start of each time slot, update the task queue based on the amount of newly arrived tasks. The channel gain is updated based on the probability model and the preset state interval. ; iii) Change the state Input is sent to the main network of TS-DQN, and the output action is... ; iv) Extend the original action using action generation strategies ; v) Input the generated actions sequentially into the convex optimization module of TS-DQN, calculate the total energy consumption of this execution, and retain the action with the highest reward through the reward function. ; vi) The system based on actions The execution results update the task queue and energy queue, and save the experience. Return to the experience replay pool; vii) The target network of TS-DQN is randomly sampled from the experience pool, and its loss function is calculated relative to that of the main network to train the main network; the main network then uses its own weights... Periodically update the target network weights ; viii) Repeat steps ii) to vii) until the specified number of iterations is completed.

2. The dynamic resource allocation method for edge computing systems according to claim 1, characterized in that, The main network is a fully connected multilayer neural network. This multilayer neural network receives state information in the real-time environment and outputs the action with the highest score to provide part of the decision for the MEC system. During training, historical data accumulated in the experience replay pool is randomly sampled and the loss function is calculated in combination with the output of the target network. Finally, the stochastic gradient descent method is used to backpropagate itself to realize the entire learning process. The target network has the same fully connected structure as the main network, that is, it serves as a historical reference model for the main network. The target network periodically replaces its own weights with the weights of the main network and outputs the Q-values ​​of relevant actions during training to assist the main network in evaluating learning errors.

3. The dynamic resource allocation method for edge computing systems according to claim 1, characterized in that, The convex optimization module is designed for solving the optimal value of convex functions based on convex sets. It utilizes the property that a local optimal solution in a convex problem is necessarily a global optimal solution, and uses the second derivative or interior point method from the convexity solution toolkit for optimization to accurately solve the input convex problem.

4. The dynamic resource allocation method for edge computing systems according to claim 1, characterized in that, The system environment initialization includes: the MEC system dividing an infinitely long time domain based on discrete time slots, specifically: each time slot is composed of... Index, and the duration is Total system bandwidth Uniformly channelized into Strip channel, using Collect the channel gains of all sub-channels in the system, where: It is in the In the first time slot, responsible for the first The device transmits data to the base station. Channel gain of each uplink sub-channel; an IoT device can offload task data in parallel through multiple sub-channels, but each sub-channel can only be allocated to one device in the same time slot; At the start of each time slot, each IoT device receives a certain size of computationally intensive task and caches the task data locally. During the duration of each time slot, these cached task data are executed on the local device or remotely on the edge server according to the first-in-first-out (FIFO) principle.

5. The dynamic resource allocation method for edge computing systems according to claim 1, characterized in that, The system environment initialization includes constructing task computation time and system energy consumption models, specifically: 2.1) In time slots Phase I, from base station to device energy ,in: It is transmitted to the device The radio frequency energy, and meets the requirements , Specify the maximum transmit power of the base station for WPT. For the energy conversion efficiency of the equipment, This is the channel gain from the base station to device j; in the remaining time In this process, the device either uses the uplink wireless channel to offload the task to the edge server, or chooses not to occupy any channel and execute the task locally; 2.2) Computing equipment Local task computation time ,in: (Unit: NAT) refers to equipment. In the time slot The size of the tasks that need to be processed internally. The number of device CPU cycles required to calculate the unit NAT data in the calculation task. It is a channel allocation indicator; when its value equals 1, the system is at the [channel allocation] level. In the time slot, decide to... Sub-channels are assigned to the first This device is used for task unloading; otherwise, its value is set to 0. In the time slot The CPU frequency in the middle is Its size can be flexibly adjusted by utilizing Dynamic Voltage and Frequency Scaling (DVFS); The maximum CPU frequency that the device can achieve, satisfying the constraints. Indicator functions when conditions If satisfied, otherwise To maintain mission integrity while ensuring equipment safety Unloading its task will result in a local processing time of [time missing]. ; 2.3) Computing equipment In the time slot The total energy consumed, i.e., the energy collected by the device in Phase I for local computation and / or offloading operations of the task, specifically: the device In the time slot Total energy consumed Among them: local computing energy consumption , For the first The effective switching capacitor coefficient of a device, then the device In the time slot Local computing power When the equipment When selecting to uninstall the task and executing it remotely on the edge server, the first Transmit power used for offloading transmission in each time slot , , For transmission time, It is the noise power variance of the base station. It is assigned to the device The total bandwidth; the corresponding energy consumption for offloading is obtained. ; when It is the allocation for the execution device The server CPU frequency for the unloading task, and meets the following requirements. , For the server's CPU maximum operating capacity, the device Server task computation time ; 2.4) Calculate the energy consumption of the server when remotely executing tasks. Running devices on edge servers Task power It is the effective switching capacitor coefficient of the server.

6. The dynamic resource allocation method for edge computing systems according to claim 1, characterized in that, The system environment initialization includes building a task queue model, specifically: according to For equipment In the time slot The device will cache the newly arrived tasks; during the duration of the time slot, a portion of the cached tasks will be sent to the edge server, or performed locally on the device according to the FIFO principle; when For equipment In the time slot The length of the task queue, where: This refers to the capacity of the buffer; the buffer queue receives newly arriving tasks in each time slot and outputs a portion of the stored tasks for execution, i.e.: Due to the causal relationship, the inequality is satisfied. .

7. The dynamic resource allocation method for edge computing systems according to claim 1, characterized in that, The system environment initialization includes an optimization problem of joint resource configuration and task scheduling: minimizing the time-averaged energy requirement of the WPT-MEC system in an infinite time domain, while ensuring the causal relationship between energy and tasks for each device, i.e., satisfying: 1) the energy consumption of a device when offloading and executing tasks locally cannot exceed the effective value; 2) the planned task size in each time slot cannot exceed the current pending task size, and these task volumes can be maintained throughout the time slot duration. The internal processing was successful, resulting in the optimization problem as follows: ,in: , The number of time slots represents the base station's transmit power used for WPT. The size of the task to be processed Channel allocation flags used for offloading tasks Uninstallation duration The CPU operating frequency of the base station and the CPU operating frequency of IoT devices .

Images