Task unloading and resource allocation method based on deep reinforcement learning in heterogeneous environment

Abstract

The invention relates to a task unloading and resource allocation method based on deep reinforcement learning in a heterogeneous environment, and the method comprises the steps: constructing an edge calculation structure of heterogeneous equipment, and randomly setting an edge server and heterogeneous wireless equipment according to the edge calculation structure; acquiring environment information, and processing the environment information; the processing comprises local equipment processing or unloading to an edge server processing through heterogeneous wireless equipment; the method comprises the following steps: acquiring data volume and consumed energy processed by local equipment and an edge server, and establishing an optimal scheduling model in combination with a resource allocation condition and an unloading scheduling condition; converting the optimal scheduling model into a Markov decision model, setting a state space, an action space and a reward function in a Markov decision process, solving the Markov decision process based on a deep reinforcement learning model, and obtaining a target decision of dynamic resource allocation and task unloading; the deep reinforcement learning model is obtained by training the actor module and the commentator module through the training set.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to a task offloading and resource allocation method based on deep reinforcement learning in a heterogeneous environment. Background Art

[0002] The development of information technology and artificial intelligence has driven the widespread adoption of mobile devices and the widespread deployment of IoT devices. It is estimated that by 2030, the global installed base of IoT devices will exceed 82 billion. The exponential growth in the number of terminals has generated a demand for processing massive amounts of access services. Emerging services have placed higher demands on low-latency, highly reliable service quality. In traditional computing models, data must be uploaded to a central server for processing. This not only causes data transmission delays but also wastes significant computing resources. Technological advances have led to the implementation of distributed data processing, such as in today's cloud computing model. However, while cloud computing has achieved a certain degree of distributed processing, data still needs to be processed on cloud servers, resulting in certain latency and efficiency issues in data transmission and processing.

[0003] To overcome these problems, alternative computing models are used to obtain processing resources closer to sensors and user devices, and to use some of these resources for data computing. This has driven the emergence of a decentralized computing paradigm. Edge computing networks, by pushing computing and storage resources to the edge close to the terminal, solve the high latency problem of traditional centralized methods, thereby meeting the low latency requirements of emerging businesses. However, most current research focuses on task latency and energy costs, and these performance indicators cannot meet the needs of sensitive tasks because such tasks have high requirements for the freshness of information. For example, in autonomous driving scenarios, these sensor devices can continuously obtain data from the surrounding environment. If the timeliness of the information received by the sensor is low, the intelligent vehicle will not be able to make correct decisions in a timely manner, resulting in serious consequences.

[0004] Therefore, when considering task offloading and resource allocation, it is necessary not only to focus on traditional performance metrics such as latency and energy consumption, but also to incorporate quantitative analysis of information freshness, namely, the Age of Information (AoI). This ensures that edge computing systems provide better services for time-sensitive tasks, thereby achieving higher task processing efficiency and reliability in complex dynamic environments. This optimization problem typically involves mixed-integer non-convex optimization with multivariable coupling, making finding the optimal solution challenging. Traditional optimization algorithms, when applied to computation offloading and resource allocation in edge computing networks, often require prior environmental information. However, in complex real-world scenarios, the number of devices is large and diverse, and computing resource requirements are dynamically changing. This information is difficult to fully acquire, limiting the algorithm's ability to provide optimized results in a short period of time. Different devices (such as CPUs, GPUs, or both) have significantly different computing capabilities and resource characteristics. Existing methods typically apply a unified feature processing approach to all devices, failing to fully exploit the heterogeneous characteristics of different device types. This leads to insufficient information exchange and makes it difficult to fully reflect the characteristics and synergies of heterogeneous devices. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a task offloading and resource allocation method based on deep reinforcement learning in a heterogeneous environment. By introducing a multiple attention mechanism, different weights are assigned according to the characteristics of the task (such as computational complexity, delay requirements, etc.), and the heterogeneous characteristics of the device are combined to optimize task offloading and scheduling, achieve more efficient task execution, and improve the perception and utilization efficiency of heterogeneous resources.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A task offloading and resource allocation method based on deep reinforcement learning in a heterogeneous environment, comprising:

[0008] Constructing an edge computing structure for heterogeneous devices, and randomly setting edge servers and heterogeneous wireless devices according to the edge computing structure;

[0009] Acquire environmental information and process the environmental information; the processing includes: local device processing or offloading to an edge server via a heterogeneous wireless device;

[0010] Obtaining the amount of data processed by the local device and the edge server and the energy consumed, and establishing an optimization scheduling model based on resource allocation and offload scheduling; the resources include: information age, spectrum bandwidth, transmission power, CPU frequency, and GPU frequency; the optimization scheduling model includes: an objective function for minimizing information age and its constraints;

[0011] The optimization scheduling model is converted into a Markov decision model, and the state space, action space and reward function in the Markov decision process are set. The Markov decision process is solved based on a deep reinforcement learning model to obtain the target decision of dynamic resource allocation and task offloading. The deep reinforcement learning model is obtained by training the actor module and the critic module using a training set; the training set includes: the state, action, reward and next state stored in the experience buffer.

[0012] Optionally, obtaining the amount of data processed by the local device and the energy consumption includes:

[0013] Obtaining the amount of data processed by the local device includes:

[0014]

[0015] in, represents the CPU frequency of the wireless device m with CPU in time slot t; represents the CPU frequency of the wireless device m with GPU in time slot t; κ cpu Indicates the number of CPU cycles required to process 1-bit data; κ gpu It represents the number of GPU cycles required to process 1-bit data, and Δt is the duration of each time slot;

[0016] Acquiring the lost energy processed by the local device includes:

[0017]

[0018] Among them, γ cpu , γ gpu Both are energy efficiency factors related to chip architecture.

[0019] Optionally, obtaining the amount of data processed and the energy consumed by the edge server includes:

[0020]

[0021]

[0022] Among them, r m (t) represents the transmission rate of heterogeneous wireless device m in time slot t, P i (t) represents the transmission power of heterogeneous wireless device m in time slot t; Δt represents the duration of each time slot.

[0023] Optionally, the transmission rate is expressed as:

[0024]

[0025] Among them, Wm P represents the spectrum bandwidth allocated to the heterogeneous wireless device m in time slot t; m (t) represents the transmission power of heterogeneous wireless device m in time slot t; h m (t) represents the channel gain between the heterogeneous wireless device m and the edge server at time slot t; σ m represents the noise power of heterogeneous wireless device m.

[0026] Alternatively, the objective function to minimize the information age is expressed as:

[0027]

[0028] Among them, a m (t) represents the information age of the sensor task; T represents the time slot; W m (t) Spectrum bandwidth of wireless device m in time slot t; represents the CPU frequency of the wireless device m with CPU in time slot t; P represents the CPU frequency of the wireless device m with GPU in time slot t; m (t) represents the transmission power of wireless device m in time slot t, and N represents the total number of wireless devices.

[0029] Optionally, the constraint condition is expressed as:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] in, represents the amount of data processed locally by wireless device m in time slot t; represents the amount of data unloaded from wireless device m to the edge server in time slot t; E represents the amount of data remaining in the task queue of wireless device m in time slot t; m represents the energy consumption of offloading the computational tasks of wireless device m to the edge server for processing; represents the maximum energy that can be allocated to wireless device m; W m(t) Spectrum bandwidth of wireless device m in time slot t; represents the maximum spectrum bandwidth that can be allocated to wireless device m; represents the CPU frequency of the wireless device m with CPU in time slot t; Indicates the maximum CPU frequency that can be allocated to wireless device m; represents the CPU frequency of wireless device m with GPU in time slot t; Indicates the maximum GPU frequency that can be assigned to wireless device m; P m (t) represents the transmission power of wireless device m in time slot t; represents the maximum transmission power that can be allocated to wireless device m, Indicates the total number of wireless devices; Indicates the number of time slots.

[0038] Optionally, setting the state space, action space, and reward function in the Markov decision process includes:

[0039] Setting the state space in the Markov decision process includes setting the task information, queue information, channel gain, and information age of all wireless devices:

[0040]

[0041] in, is the state space set, Calculate the remaining task amount for wireless device m at time slot t, a m (t) is the information age of wireless device m at time slot t, q m (t) is the task queue of wireless device m in time slot t, h m (t) is the channel gain of wireless device m at time slot t;

[0042] The action space in the Markov decision process is set to include: CPU frequency, GPU frequency, spectrum bandwidth, and energy loss;

[0043]

[0044] Among them, W m is the spectrum bandwidth, P m is the energy loss, is the action space set;

[0045] The reward function is defined as the negative of the total AoI in each time slot, and the formula is as follows:

[0046]

[0047] Among them, a m(t) is the information age of wireless device m at time slot t.

[0048] Optionally, training the deep reinforcement learning model includes:

[0049] Obtain the state received by the sensors of heterogeneous environment devices, input the state into the actor module, obtain the corresponding action, and introduce OU noise when executing the action to obtain the next state and reward;

[0050] The state, action, reward, and next state are stored in an experience buffer. The experience buffer uses a sampling method of prioritized experience replay, measures the priority of experience based on a temporal difference metric, and uses heterogeneous post-decision state learning to divide the environment into a known part and an unknown part. Target experience of the unknown part is extracted as a training set to train the actor module and the critic module.

[0051] The training set is input into the critic module. Based on several multiple attention mechanisms in the critic module, the input state features in the training set are divided into: CPU device type features, GPU device type features, and CPU-GPU device type features. The key characteristics of the heterogeneous environment device types are extracted, and learnable weights are obtained for weighted fusion. The weights of the heterogeneous environment devices are adjusted, the key features are merged, and a scalar value is output to represent the value estimate of the next state predicted by the actor module. According to the scalar value, the actor module is updated to obtain the optimal actor module.

[0052] Optionally, obtaining the target decision includes:

[0053] The state received by the sensor of the current heterogeneous environment device is obtained, the current state is input into the optimal actor module, the action corresponding to the current state is obtained, and the target decision at the current moment is obtained according to the action corresponding to the current state.

[0054] The beneficial effects of the present invention are:

[0055] This paper constructs a more realistic modeling method for heterogeneous device edge computing environments. By randomly placing edge servers and intelligent heterogeneous wireless devices, the model's versatility and applicability are enhanced. The method distinguishes the processing capabilities and energy consumption characteristics of intelligent heterogeneous wireless devices and designs adaptive resource allocation solutions for heterogeneous devices.

[0056] This invention introduces a heterogeneous post-decision state to shorten the time it takes for reinforcement learning to explore the characteristics of heterogeneous devices. It also combines a multi-attention mechanism to assign different weights based on task characteristics (such as computational complexity and latency requirements). By optimizing information age, it ensures real-time data between edge devices and servers, improving the system's responsiveness to dynamic changes. By optimizing energy consumption, it avoids wasting device resources, extends device uptime, and reduces the costs and environmental pressures associated with frequent device replacements. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 This is a flow chart of a method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to an embodiment of the present invention;

[0059] Figure 2 This is an edge computing scenario diagram of an embodiment of the present invention;

[0060] Figure 3 This is a framework diagram of the improved DDPG method according to an embodiment of the present invention;

[0061] Figure 4 This is a neural network architecture diagram of the actor module in the DDPG method framework of an embodiment of the present invention;

[0062] Figure 5 This is a neural network architecture diagram of the critic module in the DDPG method framework of an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1As shown, this embodiment discloses a task offloading and resource allocation method based on deep reinforcement learning in a heterogeneous environment, including: constructing an edge computing structure of heterogeneous devices, and randomly setting edge servers and heterogeneous wireless devices according to the edge computing structure; obtaining environmental information and processing the environmental information; processing includes: local device processing or offloading to the edge server for processing through heterogeneous wireless devices; obtaining the amount of data processed by the local device and the edge server and the energy lost, and establishing an optimization scheduling model based on the resource allocation and offloading scheduling; resources include: information age, spectrum bandwidth, transmission power, CPU frequency and GPU frequency; the optimization scheduling model includes: an objective function and its constraints for minimizing information age; converting the optimization scheduling model into a Markov decision model, setting the state space, action space and reward function in the Markov decision process, solving the Markov decision process based on the deep reinforcement learning model, and obtaining the target decision of dynamic resource allocation and task offloading; the deep reinforcement learning model uses the training set to train the actor module and the critic module; the training set includes: the state, action, reward and next state stored in the experience buffer.

[0066] Specifically: This embodiment discloses a method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment, including:

[0067] S1: Build an edge computing system with heterogeneous devices and randomly place edge servers and intelligent heterogeneous wireless devices;

[0068] S2: Build a computing model based on the edge computing system environment of heterogeneous devices;

[0069] S3: Based on the constructed computing model and information age, we optimize resource allocation such as spectrum bandwidth, transmission power, CPU frequency, and GPU frequency, as well as task offloading scheduling between heterogeneous devices and edge servers. We then transform the above optimization problems into Markov decision processes.

[0070] S4: A deep reinforcement learning method combining heterogeneous decision-making post-state learning with multiple attention mechanisms is adopted to finally obtain an intelligent decision-making strategy for dynamic resource allocation and task offloading of the edge computing system of intelligent heterogeneous devices. Based on the intelligent decision-making strategy for dynamic resource allocation and task offloading, the information age and energy consumption optimization of the heterogeneous edge environment are realized.

[0071] In S1, the process of building an edge computing system with heterogeneous devices and randomly placing edge servers and heterogeneous wireless devices is as follows:

[0072] Build a two-layer "edge-end" structure, consisting of an edge server and N intelligent heterogeneous wireless devices. Intelligent wireless devices include devices with only CPU or GPU and devices with both CPU and GPU.

[0073] An M×M coordinate system is constructed, the edge server is placed at the center of the coordinate system, and N intelligent heterogeneous wireless devices are randomly placed in the M×M coordinate system.

[0074] In S2, a computing model is built based on the edge computing system environment of heterogeneous devices, and all channels follow quasi-static flat fading. That is, in each time slot t, channel conditions such as path loss, fading effect, etc. can be regarded as fixed, but may change between different time slots. Since the channel gain is considered constant within a single time slot, the system does not need to frequently re-evaluate the channel status within the time slot.

[0075] Local processing: Due to the limited battery capacity of terminal devices, each terminal device should rationally allocate its energy for local processing and data offloading. To control the energy consumption of local processing, this embodiment uses dynamic voltage and frequency scaling to dynamically adjust the operating frequency of the processor of the intelligent heterogeneous wireless device. The amount of data processed locally in time slot t is:

[0076]

[0077] in, represents the CPU frequency of the wireless device m with CPU in time slot t; represents the CPU frequency of the wireless device m with GPU in time slot t; κ cpu Indicates the number of CPU cycles required to process 1-bit data; κ gpu It represents the number of GPU cycles required to process 1-bit data.

[0078]

[0079] Among them, the energy loss and frequency of each CPU and GPU cycle can be expressed as and γ cpu and γ gpu Represents the energy efficiency factor related to the chip architecture.

[0080] Orthogonal Frequency Division Multiple Access (OFDMA) technology is used, which combines the advantages of Orthogonal Frequency Division Multiplexing (OFDM) and multiple access technology. The channels do not interfere with each other in frequency and have independent frequency characteristics. These sub-channels are orthogonal in frequency and do not interfere with each other. According to Shannon's formula, the transmission rate of wireless devices can be expressed as:

[0081]

[0082] Among them, W m P represents the spectrum bandwidth allocated to the wireless device in time slot t; m (t) represents the transmission power of wireless device m in time slot t; h m (t) represents the channel gain between wireless device m and the edge server at time slot t; σ m represents the noise power of wireless device m.

[0083] The amount of data offloaded from the wireless device to the edge server at time slot t is:

[0084]

[0085] Among them, r m (t) represents the transmission rate of wireless device m in time slot t; Δt represents the duration of each time slot.

[0086] The energy consumed by offloading the task from the wireless device to the edge server at time slot t is:

[0087]

[0088] Among them, P i (t) represents the transmission power of wireless device m in time slot t; Δt represents the duration of each time slot.

[0089] The relationship between transmission power and energy consumption is:

[0090]

[0091] in, W represents the amount of data unloaded by wireless device m in time slot t; m (t) represents the spectrum bandwidth allocated to the wireless device at time slot t; σ m represents the noise power of wireless device m; h m (t) represents the channel gain between wireless device m and the edge server at time slot t; Δt represents the duration of each time slot.

[0092] Furthermore, in S3, based on the constructed computing model and AOI, resource allocation such as spectrum bandwidth, transmission power, CPU frequency, and GPU frequency, as well as task offloading scheduling between heterogeneous devices and edge servers, is optimized. The process of converting the above optimization problems into Markov decision processes is as follows:

[0093] The objective function of resource allocation optimization problem including AOI, spectrum bandwidth, transmission power, CPU frequency and GPU frequency is expressed as follows:

[0094] The objective function of minimizing information age is expressed as:

[0095]

[0096] Among them, a m (t) represents the information age of the sensor task; T represents the time slot; W m (t) Spectrum bandwidth of wireless device m in time slot t; represents the CPU frequency of the wireless device m with CPU in time slot t; P represents the CPU frequency of the wireless device m with GPU in time slot t; m (t) represents the transmission power of wireless device m in time slot t, and N represents the total number of wireless devices.

[0097] The constraints are expressed as:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105] in, represents the amount of data processed locally by wireless device m in time slot t; represents the amount of data unloaded from wireless device m to the edge server in time slot t; E represents the amount of data remaining in the task queue of wireless device m in time slot t; m represents the energy consumption of offloading the computational tasks of wireless device m to the edge server for processing; represents the maximum energy that can be allocated to wireless device m; W m (t) Spectrum bandwidth of wireless device m in time slot t; represents the maximum spectrum bandwidth that can be allocated to wireless device m; represents the CPU frequency of the wireless device m with CPU in time slot t; Indicates the maximum CPU frequency that can be allocated to wireless device m; represents the CPU frequency of wireless device m with GPU in time slot t; Indicates the maximum GPU frequency that can be assigned to wireless device m; P m (t) represents the transmission power of wireless device m in time slot t; P m max (t) represents the maximum transmission power that can be allocated to wireless device m.

[0106] The process of converting the above optimization problem into a Markov decision process problem is as follows:

[0107] State space: includes task information, queue information, channel gain, and information age of all wireless devices;

[0108]

[0109] Action space: including CPU frequency, GPU frequency, spectrum bandwidth W m , energy loss;

[0110]

[0111] The reward function is defined as the negative of the total AoI in each time slot, and the formula is as follows:

[0112]

[0113] Among them, a m (t) is the information age of wireless device m at time slot t.

[0114] Furthermore, training a deep reinforcement learning model involves:

[0115] Obtain the state received by the sensors of heterogeneous environment devices, input the state into the actor module, obtain the corresponding action, and introduce OU noise in the execution of the action to obtain the next state and reward;

[0116] The state, action, reward, and next state are stored in the experience buffer. The experience buffer uses a sampling method called prioritized experience replay, measures the priority of experience based on a temporal difference metric, and uses heterogeneous post-decision state learning to divide the environment into known and unknown parts. Target experience from the unknown part is extracted as a training set to train the actor module and the critic module.

[0117] The training set is input into the critic module. Based on several multiple attention mechanisms in the critic module, the input state features in the training set are divided into: CPU device type features, GPU device type features, and CPU-GPU device type features. The key characteristics of the heterogeneous environment device types are extracted, and learnable weights are obtained for weighted fusion. The weights of the heterogeneous environment devices are adjusted, the key features are merged, and a scalar value is output to represent the value estimate of the next state predicted by the actor module. Based on the scalar value, the actor module is updated to obtain the optimal actor module.

[0118] Specifically:

[0119] Step 1. Initialization;

[0120] Step 2. The environment provides a current state s, which includes the amount of remaining data, information age, data queue length, and channel gain.

[0121] Step 3. Under the current state s, the actor network is used to generate an action (strategy) a(t), i.e., bandwidth, transmission power, CPU frequency, GPU frequency, and the critic network uses terms to evaluate the value of the action, i.e., the cost v of taking action a under the current state s;

[0122] Step 4. After the action is executed, the system calculates the amount of local processing data and offloaded data for each device based on the current resource allocation and channel status;

[0123] Step 5. Update the experience buffer using the priority experience playback;

[0124] Step 6. Randomly sample a small batch of experience tuples from the experience replay buffer to train the network;

[0125] Step 7. Update the parameters of the critic network and the actor network;

[0126] Step 8. Update the cost v;

[0127] Step 9. Update the parameters of the target critic network and the target actor network;

[0128] Step 10. Repeat steps 2-9 until the algorithm converges.

[0129] Furthermore, obtaining target decisions includes:

[0130] Obtain the state received by the sensors of the current heterogeneous environment equipment, input the current state into the optimal actor module, obtain the action corresponding to the current state, and obtain the target decision at the current moment based on the action corresponding to the current state.

[0131] Specifically:

[0132] Furthermore, the deep reinforcement learning algorithm in S4 is based on the improved DDPG algorithm to solve the Markov decision process problem. The dynamic resource allocation and task offloading intelligent decision-making strategy process of the edge computing system of intelligent heterogeneous devices is obtained as follows:

[0133] Initialize two modules: θ μ and θ Q Randomly initialize the actor module μ(s|θ μ ) and the critic module Q(s,a|θ Q ); initialize the target network weight parameters Q' and μ'; clear the experience replay buffer R, set the average reward, target value, and maximum number of training steps; the average reward is the average cost of the system in the long run (information age and energy consumption), which is used to evaluate long-term performance and balance AoI and energy consumption; and the reward R(t) is the immediate reward, reflecting the performance of the current time step (information age), which is used for immediate feedback and strategy optimization.

[0134] Average reward:

[0135]

[0136] where Cost(s(t),a(t)) is the instantaneous cost at time slot t; T is the number of time slots.

[0137]

[0138] Among them, a m (s,a) represents the information age of device m; E m (s,a) represents the energy consumption of device m; Indicates the maximum energy consumption value that can be allocated to wireless device m.

[0139] Time slot t loop:

[0140] Use the actor module to take corresponding actions based on the state received by the sensors of the current heterogeneous environment equipment; introduce OU noise in the execution of the action, and then obtain feedback of random events, including the next state and the average reward v;

[0141] Among them, the actor network generates action a according to the current state s, which is the resource allocation decision of the device (such as GPU frequency, CPU frequency, transmission power, and bandwidth allocation). By continuously updating the parameters of the actor network and the critic network, the algorithm gradually optimizes the strategy so that it can minimize the AoI and meet the energy consumption constraints in the long run.

[0142] Store the current state, action, cost, and next state in the experience buffer;

[0143] The experience buffer of the present invention uses a sampling method of priority experience playback, and measures the priority of experience based on the time difference index. For new samples, when the time difference is unknown, the sample is assigned the maximum priority to ensure that the sample will be sampled at least once. A small batch of experience is extracted from the experience buffer for training; the probability of each sample is defined as:

[0144]

[0145] The present invention uses heterogeneous decision-making state learning to divide the environment into a known part and an unknown part. The transition probabilities of these two parts are and Integrated into Determined by the current state and action is called deterministic dynamics, which can be Use function f k (s,α) represents that the dynamics determined by random events are called random dynamics. If the dynamics in part of the environment are known, then only the dynamics of the unknown part need to be learned. Avoiding repeated learning of known parts, thereby increasing the rate at which the agent explores the environment;

[0146] Define the optimal decision-making state learning value function

[0147]

[0148] The present invention introduces three multiple attention mechanisms in the critic module for the characteristics of three heterogeneous devices, dividing the input state features into the characteristics of three device types: CPU, GPU, and CPU-GPU. For each device type, unique computing resources and task execution features are extracted. The attention mechanism based on device characteristics: CPU attention mechanism: focuses on capturing the task characteristics of the CPU and its relationship with other tasks. GPU attention mechanism: is used to mine important information related to the computing power and characteristics of the GPU. CPU-GPU hybrid attention mechanism: uses a specific attention mechanism to process devices that contain both CPU and GPU, extracting collaborative computing characteristics:

[0149]

[0150] Q represents the task pair list that finds the most relevant information for the current smart wireless heterogeneous device. K helps Q determine which information is most relevant and establishes a connection. V represents the actual information required from the input. Attention weights are applied to the value vector to generate the final output. It is a scaling factor used to prevent the dot product value from being too large and keep the value stable; softmax normalizes the correlation score into a probability distribution.

[0151] Finally, the outputs of the three attention mechanisms are weightedly fused through learnable weights, giving the network freedom and enabling it to dynamically adjust the weights of different device types. This weighted fusion mechanism enhances the flexibility and performance of the model in heterogeneous device scenarios.

[0152] Furthermore, update the loss value of the critic module:

[0153]

[0154] The loss function is used to update the parameters of the critic network, Represented as a given state and The critic network's estimate of the value of the state. Represents the target value; by minimizing the loss function Critic module parameters It is continuously optimized so that the critic module can estimate the policy value more accurately.

[0155] Policy gradient updates the actor policy network:

[0156]

[0157] in, Gradient of the actor module; μ(s|θ μ ) represents the output of the actor network, in state s, based on the parameter θ μ the actions taken; Gradient of the critic module; Q(s,a|θ Q ) represents the output of the critic network, which represents the Q value when the state s and action a are given; a = μ(s) means that when calculating the gradient of the critic network, the action μ(s) given by the actor network needs to be used.

[0158] Update average reward:

[0159] v t+1 =(1-β)v t +β(s(t),α(t));

[0160] Among them, the average reward is updated by the time difference method, and β is a smoothing factor; it guides policy optimization and helps the actor network generate a strategy that can minimize AoI and meet energy consumption constraints.

[0161] Update target network: Use the soft update method to update the target network parameters:

[0162] θ Q′ ←τθ Q +(1-τ)θ Q′ ;

[0163] θ μ′ ←τθ μ +(1-τ)θ μ′ ;

[0164] Among them, θ Q represents the parameters of the critic module; θ Q′ represents the parameters of the target critic module; θ μ represents the actor module; θ μ′ represents the target actor module; τ is a constant, usually with a small value, τ<<1, which is used to control the update speed of the target network.

[0165] like Figure 1 As shown, this embodiment discloses a method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment, including:

[0166] S1: Build an edge computing system with heterogeneous devices and randomly place edge servers and intelligent heterogeneous wireless devices;

[0167] Build a two-layer "edge-end" structure, consisting of an edge server and N intelligent heterogeneous wireless devices. Intelligent wireless devices include devices with only CPU or GPU and devices with both CPU and GPU.

[0168] An N×N coordinate system is constructed, the edge server is placed at the center of the coordinate system, and N intelligent heterogeneous wireless devices are randomly placed in the N×N coordinate system.

[0169] S2: Build a computing model based on the edge computing system environment of heterogeneous devices;

[0170] like Figure 2 The base station possesses computing resources. Devices observe the surrounding environment and generate computing tasks containing relevant raw data when the environment changes. Environmental information can only be obtained after data processing. This embodiment focuses on applications that require sequential data processing, meaning that the processing of one task depends on the computational results of previous tasks. An illustrative example is time series anomaly detection, where the identification of abnormal patterns relies on historical environmental information. Therefore, tasks on each UD must be processed on a first-come, first-served (FCFS) basis. It is assumed that each task can be further divided into subtasks, and each subtask can be freely processed by the local device or offloaded to the edge server via wireless transmission. This offloading method is called "partial offloading." Regardless of whether the generated task is processed at the edge or the terminal, the computation results are synchronized between the edge and the terminal to confirm the processing of the next task. The amount of computational results is typically small, so the time required for dual-end synchronization and the corresponding energy loss are negligible.

[0171] Local processing: Due to the limited battery capacity of terminal devices, each terminal device should reasonably allocate its energy for local processing and data offloading. To control the energy consumption of local processing, this embodiment uses dynamic voltage and frequency scaling (DVFS) to dynamically adjust the operating frequency of the processor of the intelligent heterogeneous wireless device. Assume f m (t) is the CPU and GPU frequency of wireless device m in time slot t, κ cpu and κ gpu The number of CPU and GPU cycles required to process one bit of data is:

[0172]

[0173] The energy loss and frequency of each CPU and GPU cycle can be expressed as and where γ cpu and γ gpu is an energy efficiency factor related to the chip architecture. The energy loss of local calculation is as follows:

[0174]

[0175] 2) Computational Offloading: This embodiment assumes the use of Orthogonal Frequency Division Multiple Access (OFDMA) technology, which combines the advantages of Orthogonal Frequency Division Multiplexing (OFDM) and Multiple Access technologies. Channels do not interfere with each other in frequency and have independent frequency characteristics. These sub-channels are orthogonal in frequency and do not interfere with each other. According to Shannon's formula, the transmission rate can be expressed as:

[0176]

[0177] Among them, W m (t) is the time slot assigned to UD i The spectrum bandwidth, P m (t) is the transmission task power of the intelligent heterogeneous wireless device m, h m (t) is the channel gain between the smart heterogeneous wireless device m and the edge server, σ m represents the noise power of the intelligent heterogeneous wireless device m.

[0178] To simplify the problem, all channels are assumed to follow quasi-static flat fading. That is, within each time slot, channel conditions such as path loss and fading effects can be considered fixed, but may change between different time slots. Since the channel gain is considered constant within a single time slot, the system does not need to frequently re-evaluate the channel state within the time slot. It is also assumed that the computing power of the edge server is stronger than that of the terminal device, and the computational data offloaded to the edge server can be processed within a single time slot.

[0179] The amount of data and energy consumed by unloading from the intelligent heterogeneous wireless device m to the edge server at time slot t is:

[0180]

[0181]

[0182] The above formulas can be combined:

[0183]

[0184] S3: Based on the constructed computing model and information age, we optimize resource allocation such as spectrum bandwidth, transmission power, CPU frequency, and GPU frequency, as well as task offloading scheduling between heterogeneous devices and edge servers. We then transform the above optimization problems into Markov decision processes.

[0185] The objective function is expressed as follows:

[0186]

[0187] Constraints:

[0188]

[0189]

[0190]

[0191]

[0192]

[0193]

[0194]

[0195] Among them, a m (t) represents the information age of the sensor task; T represents the time slot; represents the amount of data processed locally by wireless device m in time slot t; represents the amount of data unloaded from wireless device m to the edge server in time slot t; E represents the amount of data remaining in the task queue of wireless device m in time slot t; m represents the energy consumption of offloading the computational tasks of wireless device m to the edge server for processing; represents the maximum energy that can be allocated to wireless device m; W m (t) Spectrum bandwidth of wireless device m in time slot t; represents the maximum spectrum bandwidth that can be allocated to wireless device m; represents the CPU frequency of the wireless device m with CPU in time slot t; Indicates the maximum CPU frequency that can be allocated to wireless device m; represents the CPU frequency of wireless device m with GPU in time slot t; Indicates the maximum GPU frequency that can be assigned to wireless device m; P m (t) represents the transmission power of wireless device m in time slot t; P m max (t) represents the maximum transmission power that can be allocated to wireless device m.

[0196] S4: A deep reinforcement learning method combining heterogeneous decision-making post-state learning with multiple attention mechanisms is adopted to finally obtain an intelligent decision-making strategy for dynamic resource allocation and task offloading of the edge computing system of intelligent heterogeneous devices. Based on the intelligent decision-making strategy for dynamic resource allocation and task offloading, the information age and energy consumption optimization of the heterogeneous edge environment are realized.

[0197] State space: includes task information, queue information, channel gain, and information age of all wireless devices;

[0198]

[0199] Action space: including CPU frequency, GPU frequency, spectrum bandwidth W m , energy loss;

[0200]

[0201] The reward function is defined as the negative of the total AoI in each time slot, and the formula is as follows:

[0202]

[0203] Furthermore, the deep reinforcement learning algorithm in S4 is based on the improved DDPG algorithm to solve the Markov decision process problem, such as Figure 3 The dynamic resource allocation and task offloading intelligent decision-making process of the edge computing system of intelligent heterogeneous devices is obtained as follows:

[0204] Initialize two modules: θ μ and θ Q Randomly initialize the actor module μ(s|θ μ ) and the critic module Q(s,a|θ Q ); Initialize the target network weight parameters Q' and μ'; Clear the experience replay buffer R, set the average reward, target value, and maximum number of training steps;

[0205] Time slot t loop:

[0206] Use the actor module to take corresponding actions based on the status received by the sensors of the current heterogeneous environment devices; introduce OU noise during the execution of the action, and then obtain feedback from random events, including the next state and cost;

[0207] Store the current state, action, cost, and next state in the experience buffer;

[0208] The experience buffer of the present invention uses a sampling method of priority experience playback, and measures the priority of experience based on the time difference index. For new samples, when the time difference is unknown, the sample is assigned the maximum priority to ensure that the sample will be sampled at least once. A small batch of experience is extracted from the experience buffer for training; the probability of each sample is defined as:

[0209]

[0210] The present invention uses heterogeneous decision-making state learning to divide the environment into a known part and an unknown part. The transition probabilities of these two parts are and Integrated into Determined by the current state and action is called deterministic dynamics, which can be Use function f k (s,α) represents that the dynamics determined by random events are called random dynamics. If the dynamics in part of the environment are known, then only the dynamics of the unknown part need to be learned. Avoiding repeated learning of known parts, thereby increasing the rate at which the agent explores the environment;

[0211] Define the optimal decision-making state learning value function

[0212]

[0213] In the critic module, the present invention introduces three multiple attention mechanisms for the characteristics of three heterogeneous devices, divides the input state features into the characteristics of three device types: CPU, GPU, and CPU-GPU, and extracts unique computing resources and task execution features for each device type. The attention mechanism based on device characteristics: CPU attention mechanism: focuses on capturing the task characteristics of the CPU and its relationship with other tasks. GPU attention mechanism: is used to mine important information related to the computing power and characteristics of the GPU. CPU-GPU hybrid attention mechanism: uses a specific attention mechanism to process devices that contain both CPU and GPU to extract collaborative computing characteristics.

[0214]

[0215] Q represents the task pair list that searches for the most relevant information for the current smart wireless heterogeneous device. K helps Q determine which information is most relevant and establishes a connection. V represents the actual information required from the input. Attention weights are applied to the value vector to generate the final output. It is a scaling factor used to prevent the dot product value from being too large and keep the value stable; softmax normalizes the correlation score into a probability distribution.

[0216] Finally, the outputs of the three attention mechanisms are passed through the learnable weights w = [w cpu ,w gpu ,w cpu-gpu ] performs weighted fusion, giving the network freedom so that it can dynamically adjust the weights of different device types. This weighted fusion mechanism enhances the flexibility and performance of the model in heterogeneous device scenarios.

[0217] Furthermore, update the loss value of the critic module:

[0218]

[0219] Policy gradient updates the actor policy network:

[0220]

[0221] Update average reward:

[0222] v=(1-β)v+β(s(t),α(t));

[0223] Where β is the smoothing factor;

[0224] Update target network: Use the soft update method to update the target network parameters:

[0225] θ Q′ ←τθ Q +(1-τ)θ Q′ ;

[0226] θ μ′ ←τθ μ +(1-τ)θ μ′ ;

[0227] Figure 4-Figure 5 It is the neural network architecture of the actor module and critic module in the DDPG network;

[0228] The actor network module receives the state of the edge environment and generates corresponding actions. The system state is composed of the individual states of each heterogeneous device. The state of each device is represented as a four-tuple containing four features, resulting in a system state dimension of 4×N. N represents the number of heterogeneous devices. The system state is first input into a fully connected layer consisting of 128 neurons, each of which uses a Rectified Linear Unit (ReLU) activation function for nonlinear transformation. The ReLU output is normalized through a batch normalization layer (Batch Normalization) to improve training stability and accelerate the convergence of the deep neural network (DNN). The normalized output is then input into another fully connected layer with a leaky ReLU activation function and a batch normalization layer. The final layer of the actor network is a fully connected layer with 4×N neurons, corresponding to the four continuous actions of each wireless device (e.g., CPU frequency, GPU frequency, transmit power, and wireless bandwidth). Finally, a sigmoid activation function is used to ensure that the output falls within the interval [0, 1].

[0229] The architecture of the critic network differs from that of the actor network. The critic network input has the same shape as the actor network input. The present invention then divides the input into three independent feature branches based on the characteristics of each device type (CPU, GPU, or CPU-GPU). Feature masks are generated by heterogeneous device types to filter the input features. A dedicated multi-head attention mechanism is introduced for each device type to extract key characteristics specific to the device type. The three attention outputs are then weighted and fused using learnable weights, enabling dynamic adjustment to accommodate varying device ratios and task requirements. The merged features are flattened and then fed into a fully connected layer. The first layer uses a Reluctant Unit (ReLU) activation function; the second layer uses a linear activation function, followed by a Leaky Reluctant Unit (Leaky ReLU) to enhance nonlinear modeling capabilities. Finally, a fully connected layer outputs a scalar value representing the value assessment of the input state-action pair. To train these two deep neural networks, two Adam optimizers were used, with learning rates of 0.001 (for the actor network) to ensure smoother policy updates and 0.002 (for the critic network), slightly faster than the learning rate in the actor network to facilitate faster adjustment of value assessments.

[0230] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A task offloading and resource allocation method based on deep reinforcement learning in a heterogeneous environment, characterized by: include: Constructing an edge computing structure for heterogeneous devices, and randomly setting edge servers and heterogeneous wireless devices according to the edge computing structure; Acquire environmental information and process the environmental information; the processing includes: local device processing or offloading to an edge server via a heterogeneous wireless device; Obtaining the amount of data processed by the local device and the edge server and the energy consumed, and establishing an optimization scheduling model based on resource allocation and offload scheduling; the resources include: information age, spectrum bandwidth, transmission power, CPU frequency, and GPU frequency; the optimization scheduling model includes: an objective function for minimizing information age and its constraints; The optimization scheduling model is converted into a Markov decision model, and the state space, action space and reward function in the Markov decision process are set. The Markov decision process is solved based on a deep reinforcement learning model to obtain the target decision of dynamic resource allocation and task offloading. The deep reinforcement learning model is obtained by training the actor module and the critic module using a training set; the training set includes: the state, action, reward and next state stored in the experience buffer.

2. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 1 is characterized in that: Obtaining the amount of data processed by the local device and the energy consumption includes: Obtaining the amount of data processed by the local device includes: in, represents the CPU frequency of wireless device m with CPU in time slot t; represents the CPU frequency of the wireless device m with GPU in time slot t; κ cpu Indicates the number of CPU cycles required to process 1-bit data; κ gpu It represents the number of GPU cycles required to process 1-bit data, and Δt is the duration of each time slot; Acquiring the lost energy processed by the local device includes: Among them, γ cpu , γ gpu Both are energy efficiency factors related to chip architecture.

3. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 1, characterized in that: Obtaining the amount of data processed by the edge server and the energy consumed includes: Among them, r m (t) represents the transmission rate of heterogeneous wireless device m in time slot t, P i (t) represents the transmission power of heterogeneous wireless device m in time slot t; Δt represents the duration of each time slot.

4. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 3 is characterized in that: The transmission rate is expressed as: Among them, W m P represents the spectrum bandwidth allocated to the heterogeneous wireless device m in time slot t; m (t) represents the transmission power of heterogeneous wireless device m in time slot t; h m (t) represents the channel gain between the heterogeneous wireless device m and the edge server at time slot t; σ m represents the noise power of heterogeneous wireless device m.

5. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 1, characterized in that: The objective function of minimizing information age is expressed as: Among them, a m (t) represents the information age of the sensor task; T represents the time slot; W m (t) Spectrum bandwidth of wireless device m in time slot t; represents the CPU frequency of wireless device m with CPU in time slot t; P represents the CPU frequency of the wireless device m with GPU in time slot t; m (t) represents the transmission power of wireless device m in time slot t, and N represents the total number of wireless devices.

6. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 1, characterized in that: The constraints are expressed as: in, represents the amount of data processed locally by wireless device m in time slot t; represents the amount of data unloaded from wireless device m to the edge server in time slot t; E represents the amount of data remaining in the task queue of wireless device m in time slot t; m represents the energy consumption of offloading the computational tasks of wireless device m to the edge server for processing; represents the maximum energy that can be allocated to wireless device m; W m (t) Spectrum bandwidth of wireless device m in time slot t; represents the maximum spectrum bandwidth that can be allocated to wireless device m; represents the CPU frequency of wireless device m with CPU in time slot t; Indicates the maximum CPU frequency that can be allocated to wireless device m; represents the CPU frequency of wireless device m with GPU in time slot t; Indicates the maximum GPU frequency that can be assigned to wireless device m; P m (t) represents the transmission power of wireless device m in time slot t; represents the maximum transmission power that can be allocated to wireless device m, Indicates the total number of wireless devices; Indicates the number of time slots.

7. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 1, characterized in that: Setting the state space, action space, and reward function in the Markov decision process involves: Setting the state space in the Markov decision process includes setting the task information, queue information, channel gain, and information age of all wireless devices: Among them, S is the state space set, Calculate the remaining task amount for wireless device m at time slot t, a m (t) is the information age of wireless device m at time slot t, q m (t) is the task queue of wireless device m in time slot t, h m (t) is the channel gain of wireless device m at time slot t; The action space in the Markov decision process is set to include: CPU frequency, GPU frequency, spectrum bandwidth, and energy loss; Among them, W m is the spectrum bandwidth, P m is the energy loss, is the action space set; The reward function is defined as the negative of the total AoI in each time slot, and the formula is as follows: Among them, a m (t) is the information age of wireless device m at time slot t.

8. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 1, characterized in that: Training the deep reinforcement learning model includes: Obtain the state received by the sensors of heterogeneous environment devices, input the state into the actor module, obtain the corresponding action, and introduce OU noise when executing the action to obtain the next state and reward; The state, action, reward, and next state are stored in an experience buffer. The experience buffer uses a sampling method of prioritized experience replay, measures the priority of experience based on a temporal difference metric, and uses heterogeneous post-decision state learning to divide the environment into a known part and an unknown part. Target experience of the unknown part is extracted as a training set to train the actor module and the critic module. The training set is input into the critic module. Based on several multiple attention mechanisms in the critic module, the input state features in the training set are divided into: CPU device type features, GPU device type features, and CPU-GPU device type features. The key characteristics of the heterogeneous environment device types are extracted, and learnable weights are obtained for weighted fusion. The weights of the heterogeneous environment devices are adjusted, the key features are merged, and a scalar value is output to represent the value estimate of the next state predicted by the actor module. According to the scalar value, the actor module is updated to obtain the optimal actor module.

9. The method for task offloading and resource allocation based on deep reinforcement learning in a heterogeneous environment according to claim 8, characterized in that: Obtaining the target decision includes: The state received by the sensor of the current heterogeneous environment device is obtained, the current state is input into the optimal actor module, the action corresponding to the current state is obtained, and the target decision at the current moment is obtained according to the action corresponding to the current state.

Images