Deep reinforcement learning calculation unloading method and system based on multi-commentator mechanism

Abstract

The invention discloses a deep reinforcement learning calculation unloading method and system based on a multi-reviewer mechanism, and the method comprises the steps: obtaining a local state vector of an equipment user, carrying out the decision making through a multi-reviewer near-end strategy optimization algorithm, and obtaining an action decision of the equipment user in a current state; data interaction with the environment is carried out, the interaction time delay and the interaction energy consumption of the equipment user are obtained, calculation is carried out according to the completion condition of data interaction, and a time delay reward and an energy consumption reward are obtained; based on a centralized training distributed execution mechanism and a multi-commentator mechanism, performing training updating on the action decision of the device user to obtain a trained device user; and based on the trained equipment user, carrying out independent decision and executing a calculation unloading task. According to the invention, intelligent agents can be helped to coordinate with each other and make decisions independently. The deep reinforcement learning calculation unloading method and system based on the multi-commentator mechanism can be widely applied to the technical field of industrial Internet of Things edge calculation.

Description

Technical Field

[0001] The present invention relates to the field of industrial Internet of Things edge computing technology, and in particular to a deep reinforcement learning computation offloading method and system based on a multi-critic mechanism. Background Art

[0002] In practical applications, industrial edge computing environments present heterogeneous network architectures with multiple devices and servers. Industrial IoT devices and edge servers compete for resources, yet they also require collaboration to complete complex computing tasks. In this environment, devices must independently decide whether to perform local computations or offload tasks to edge servers based on their task characteristics and network status. They must also rationally allocate computing resources and transmission power to meet latency constraints, reduce energy consumption, and improve task completion rates.

[0003] Traditional solutions for computational offloading and resource allocation are mostly based on rules, heuristic algorithms, or control theory. While these methods can address the problem to a certain extent, they have significant shortcomings, such as reliance on prior knowledge of the system, the need for precise modeling, and high computational complexity. For example, methods based on rules, heuristics, or control theory rely on prior knowledge and require precise modeling and numerous iterations, making them difficult to adapt to the dynamic and complex MEC environment. Methods based on deep reinforcement learning often perform multi-objective optimization by simply weighting latency and energy consumption, resulting in the same utility value for different optimization objective combinations and preventing targeted optimization. Control theory methods also suffer from poor coordination capabilities. In a hybrid environment with multiple devices and servers, devices rely only on local information and lack a global coordination mechanism. Consequently, these technologies are unable to dynamically coordinate multi-device task offloading and resource allocation, resulting in high task latency, high energy consumption, and low completion rates. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a deep reinforcement learning computation offloading method and system based on a multi-critic mechanism. Through a centralized training distributed execution mechanism and a multi-critic mechanism, it helps intelligent agents coordinate with each other and make independent decisions. At the same time, different optimization strategies can be adopted for the actor network according to different optimization effects during the training process.

[0005] The first technical solution adopted by the present invention is: a deep reinforcement learning computation offloading method based on a multi-critic mechanism, comprising the following steps:

[0006] Obtain the local state vector of the device user and make a decision through the multi-critic proximal policy optimization algorithm to obtain the action decision of the device user in the current state;

[0007] Based on the action decisions of the device user in the current state, the device user interacts with the environment to obtain data, obtain the interaction delay and interaction energy consumption of the device user, and calculate them according to the completion status of the data interaction to obtain delay rewards and energy consumption rewards;

[0008] Based on the centralized training distributed execution mechanism and multi-critic mechanism, combined with delay rewards and energy consumption rewards, the device user's action decision is trained and updated to obtain the trained device user;

[0009] Based on the trained device user, the user makes independent decisions to perform computational offloading tasks.

[0010] Furthermore, the step of obtaining the local state vector of the device user and making a decision through the multi-critic proximal strategy optimization algorithm to obtain the action decision under the current state of the device user specifically includes:

[0011] Obtain the task information and status information of the device user in the current time interval and construct a local state vector;

[0012] Based on the observation information of the environment, the local state vector is forward propagated through the actor network model to obtain the mean and standard deviation values with Gaussian distribution;

[0013] The mean and standard deviation values of the Gaussian distribution are sampled to determine the action decision under the current state of the device user.

[0014] Furthermore, the expressions of the action decisions under the current status of the users of the several devices are specifically as follows:

[0015] a m (t) = {x m (t),ch m (t), α m (t), β m (t)}

[0016] In the above formula, a m (t) represents the action decision in the current state, x m (t) indicates whether the task is computed locally or on which edge server the computation is offloaded. m (t) represents the transmission channel selected by the agent, α m (t) represents the computing resources allocated for local computing, β m (t) represents the transmission power during data transmission.

[0017] Furthermore, the step of interacting with the environment based on the action decision of the device user in the current state, obtaining the interaction delay and interaction energy consumption of the device user, and calculating the delay reward and energy consumption reward based on the completion status of the data interaction specifically includes:

[0018] Device users interact with the environment based on their current state and perform corresponding computing tasks.

[0019] If the computing task is to perform local computing, the latency and energy consumption are calculated based on the computing resources of the user device and the task requirements, and the latency and energy consumption of the local computing are determined;

[0020] If the computing task is to perform offloaded computing, determine the transmission delay of the offloaded computing, the edge server computing delay, and the transmission energy consumption of the offloaded computing based on the transmission channel and the computing capacity of the edge server;

[0021] Determine the device user interaction latency during a preset time interval by combining the local computation latency, the offloaded computation transmission latency, and the edge server computation latency.

[0022] Combine the energy consumption of local computing with the transmission energy consumption of offloaded computing to determine the interaction energy consumption of device users during preset time intervals;

[0023] According to the completion of data interaction, the interaction delay of the device user in the preset time interval and the interaction energy consumption of the device user in the preset time interval are rewarded and calculated respectively to obtain the delay reward and energy consumption reward.

[0024] Furthermore, the expression of the interaction delay of the device user in the preset time interval is specifically as follows:

[0025]

[0026] In the above formula, d m (t) represents the interaction delay between the device user and the preset time interval, Indicates the latency of local computing, represents the sum of the transmission delay of the offloaded calculation and the edge server calculation delay, x m (t) represents the indicator function;

[0027] The expression of the interaction energy consumption of the device user in the preset time interval is specifically as follows:

[0028]

[0029] In the above formula, e m (t) represents the interaction energy consumption of the device user in the preset time interval, represents the energy consumption of local computing, represents the transmission energy consumption of offloaded calculation, x m (t) represents the indicator function.

[0030] Furthermore, the calculation expressions of the delay reward and energy consumption reward are specifically as follows:

[0031]

[0032] In the above formula, represents the delayed reward, represents energy consumption reward, d m (t) represents the interaction delay between the device user and the preset time interval, e m (t) represents the interaction energy consumption of the device user in the preset time interval, p d and p e are the timeout penalty constants representing delay and energy consumption respectively.

[0033] Furthermore, the step of training and updating the action decision of the device user based on the centralized training distributed execution mechanism and the multi-critic mechanism, combined with the delay reward and the energy consumption reward, to obtain the trained device user specifically includes:

[0034] Feedback the latency reward and energy consumption reward to the user device to obtain the new status of the user device;

[0035] Combining the current state of the user device, action decision, delay reward, energy consumption reward and new state, an experience pool is constructed and stored in the trajectory collector;

[0036] Based on the multi-critic mechanism, a critic is assigned to each of the delay reward and the energy consumption reward, namely the delay critic network and the energy consumption critic network, and the loss function of the delay critic network and the loss function of the energy consumption critic network are determined;

[0037] Extract data from the trajectory collector. The delay critic network estimates the delay value function based on the delay reward and the new state. The energy critic network estimates the energy value function based on the energy reward and the new state. Minimize the loss function of the delay critic network and the loss function of the energy critic network to obtain the delay optimization effect and energy consumption optimization effect.

[0038] If the delay optimization effect is less than the energy consumption optimization effect, the policy network is updated through the delay value function. If the energy consumption optimization effect is less than the delay optimization effect, the policy network is updated through the energy consumption value function. The action decision of the device user is trained and updated to obtain the trained device.

[0039] Furthermore, the loss function expressions of the delay critic network and the energy consumption critic network are specifically as follows:

[0040]

[0041] In the above formula, represents the loss function of the delayed critic network, represents the loss function of the energy-consumption critic network, S t Represents the state of all agents in the environment at the current moment, S t+1 Represents the state of all agents in the environment at the next moment, and Represent delay reward and energy consumption reward respectively, γ represents discount factor, V d (·) and V e (·) denotes the delay critic network and energy consumption critic network, respectively.

[0042] Furthermore, the expressions of the delay value function and the energy consumption value function are specifically as follows:

[0043]

[0044] In the above formula, represents the delay value function, represents the energy consumption value function, γ represents the discount factor, and λ represents a hyperparameter used to calculate the generalized advantage estimate (GAE). They represent the delay reward and energy consumption reward obtained by the agent at time t+l, S t+l+1 and S t+l They represent the states of all agents in the environment at time t+l+1 and time t+l respectively, and l represents the index variable in the generalized advantage estimation.

[0045] The second technical solution adopted by the present invention is: a deep reinforcement learning computation offloading system based on a multi-critic mechanism, comprising:

[0046] The first module is used to obtain the local state vector of the device user and make decisions through the multi-critic proximal policy optimization algorithm to obtain the action decision of the device user in the current state;

[0047] The second module is used to interact with the environment based on the device user's current action decisions, obtain the device user's interaction delay and interaction energy consumption, and calculate them based on the completion of the data interaction to obtain delay rewards and energy consumption rewards;

[0048] The third module is used to train and update the device user's action decision based on the centralized training distributed execution mechanism and the multi-critic mechanism, combined with the delay reward and energy consumption reward, to obtain the trained device user;

[0049] The fourth module is used to make independent decisions based on trained device users to perform computational offloading tasks.

[0050] The method and system of the present invention have the following beneficial effects: by obtaining the local state vector of the device user and making decisions through a multi-critic proximal policy optimization algorithm, the present invention further interacts with the environment based on the device user's action decision in the current state, obtains the device user's interaction delay and interaction energy consumption, and calculates them separately according to the completion of the data interaction to obtain a delay reward and an energy consumption reward. Independent critic networks are designed for delay and energy consumption respectively, avoiding the policy confusion caused by traditional weighted summation. The method and system can dynamically calculate the offload in a heterogeneous industrial edge computing environment with multiple industrial IoT devices and multiple edge servers. Each device needs to compete and coordinate with each other, jointly considering offload decisions and resource allocation in a dynamic environment, thereby maximizing the total system utility and task completion rate. Finally, based on a centralized training distributed execution mechanism and a multi-critic mechanism, combined with delay rewards and energy consumption rewards, the device user's action decision is trained and updated. The intelligent agents are allowed to share information during the training phase to promote mutual coordination. The centralized training distributed execution (CTDE) mechanism and the multi-critic mechanism help the intelligent agents coordinate with each other and make independent decisions. At the same time, different optimization strategies can be adopted for the actor network based on different optimization effects during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of the steps of the deep reinforcement learning calculation offloading method based on the multi-critic mechanism of the present invention;

[0052] Figure 2 This is a structural block diagram of the deep reinforcement learning computation offloading system based on the multi-critic mechanism of the present invention;

[0053] Figure 3 is a schematic diagram of a network structure provided by a specific embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of the convergence performance of a device under different algorithms provided by a specific embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of the total utility of different numbers of devices provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0057] First, it's important to note that with the rapid development of the Industrial Internet of Things (IIoT) and smart manufacturing technologies, the demands for device connectivity and data processing in industrial production scenarios are becoming increasingly complex. IIoT devices, as key data nodes in industrial systems, are responsible for collecting, transmitting, and intelligently controlling various types of data in the production process. The tasks generated by these devices often require high real-time and precision. For example, in smart manufacturing, production equipment must respond to control commands in real time to ensure smooth production processes. In equipment monitoring scenarios, sensors must transmit data instantly to detect and address potential faults.

[0058] However, the computing power of industrial IoT devices is limited by hardware design and energy consumption, making it difficult to independently complete these computationally intensive tasks. While cloud computing demonstrates powerful data processing capabilities, it is typically deployed in large data centers far from industrial sites. The latency between data transmission to the cloud and the return of results cannot meet the stringent real-time requirements of industrial scenarios.

[0059] To address these challenges, Mobile Edge Computing (MEC) technology has emerged. MEC deploys edge servers at the edge of the network, bringing computing resources and services closer to data sources and users, significantly reducing data transmission latency. These edge servers, typically deployed near base stations, possess greater computing power than industrial IoT devices and can quickly respond to computing requests from local devices.

[0060] In practical applications, industrial edge computing environments feature heterogeneous network architectures with multiple devices and servers coexisting. Industrial IoT devices and edge servers compete for resources, yet they also require collaboration to complete complex computing tasks. In this environment, devices must autonomously decide whether to offload computations locally or to edge servers based on their task characteristics and network status, and rationally allocate computing resources and transmission power to meet latency constraints while reducing energy consumption and improving task completion rates. Traditional solutions for computation offloading and resource allocation are often based on rules, heuristic algorithms, or control theory. While these approaches can address this problem to a certain extent, they have significant limitations, such as relying on prior knowledge of the system, requiring precise modeling, and exhibiting high computational complexity. In contrast, deep reinforcement learning (DRL) technology offers unique advantages. It enables intelligent agents to learn optimal policies through interaction with the environment in complex environments. Without requiring precise pre-defined modeling, it can make dynamic decisions in real time, possesses excellent generalization capabilities, and possesses multi-objective optimization capabilities. This makes it ideally suited to addressing the dynamic changes in devices and complex task demands in industrial edge computing environments.

[0061] Based on this, an embodiment of the present invention proposes an algorithm based on deep reinforcement learning, which helps intelligent agents coordinate with each other and make independent decisions through a centralized training distributed execution (CTDE) mechanism and a multi-critic mechanism. At the same time, different optimization strategies can be adopted for the actor network according to different optimization effects during the training process.

[0062] Reference Figure 1 The present invention provides a deep reinforcement learning computation offloading method based on a multi-critic mechanism, the method comprising the following steps:

[0063] S100, obtaining the local state vector of the device user and making a decision through a multi-critic proximal strategy optimization algorithm, and obtaining an action decision of the device user in the current state;

[0064] Specifically, the task information and state information of the device user in the current time interval are obtained to construct a local state vector. Based on the observation information of the environment, the local state vector is forward propagated through the actor network model to obtain the mean and standard deviation values with Gaussian distribution. The mean and standard deviation values with Gaussian distribution are sampled to determine the action decision under the current state of the device user.

[0065] In this embodiment, each device collects information about its tasks (e.g., data size, computational density, maximum allowable latency) and its own state information (e.g., maximum computational resources, maximum transmission power) within its current time slot to form a local state vector. These local state vectors are then concatenated into a global state vector for use in the centralized training phase. In the multi-agent CTDE mechanism, the critic network is centralized, meaning it has access to observation and action information from all agents during training.

[0066] Therefore, by concatenating the local state vectors of each device into a global state vector, the centrally trained critic network obtains information about all agents in the system. This allows the critic network to globally evaluate the quality of the current joint action, providing more accurate gradient information for each agent's actor network. In contrast, if only an independent multi-agent algorithm is used, centralized training is unnecessary; only the state information of each agent is required, eliminating the need for concatenation.

[0067] Furthermore, each device selects specific actions based on its own local state vector through our proposed multi-critic proximal policy optimization algorithm, including local computing or offloading decisions of tasks, transmission channel selection, computing resource allocation ratio, and transmission power allocation ratio.

[0068] In this embodiment, during the execution phase, each agent's actor network selects actions completely independently based on its own local observations.

[0069] Specifically, when the agent receives observation information in the environment After that, it will be input into its own actor network, and after the network's forward propagation calculation, it will output the mean and standard deviation of a Gaussian distribution. By constructing a Gaussian distribution for sampling, the decision taken by the agent in this state is obtained, that is, a m (t) = {x m (t), ch m (t), α m (t), β m (t)}. x m (t) represents the task being computed locally or on which edge server the computation is offloaded. m (t) is the transmission channel selected by the agent. m (t) is the computing resource allocated for local computing. m (t) is the transmission power during data transmission.

[0070] In some specific embodiments, a heterogeneous industrial edge computing environment is constructed with multiple industrial edge computing devices, multiple edge servers, and a cloud computing center. Industrial IoT devices are devices that connect industrial systems to achieve data collection, transmission, and intelligent control. We define these devices as Each device is an agent. Similar to devices, the set of edge servers is represented as Each edge server has different computing resources and thus different computing capabilities. We assume that the edge servers are deployed near base stations (BSs) to reduce data transmission latency, so we also ignore the communication overhead between the edge servers and the base stations. In addition, the channels in the model are also heterogeneous. We define the channel set as The channel state information for each channel is completely different. Due to the limited computing power of the device and its inability to obtain information about other devices, centralized training cannot be performed locally. Computation must be performed in a cloud computing center. After training is complete, the trained agent is deployed to the corresponding device.

[0071] In this embodiment, we divide each episode into T time slots, and the time slot set is The duration of each time slot is δ seconds. The model is iterated and trained once after each round. At the beginning of each time slot t, each device will randomly generate a task. The task generated by device m in time slot t is used by G m (t) = {o m (t), c m (t), d m，max (t)} to define. m (t) (unit: bits) represents the data size of the task generated by device m in time interval t. m (t) (unit is the number of cycles required to process each bit of data, cycles / bit) represents the computational density, that is, the computing resources required to perform the task. m，max (t) (unit: s) represents the maximum processing delay allowed for the task.

[0072] The computation offloading strategy of device m for tasks generated in time interval t consists of four parts, namely a m (t) = {x m (t),ch m (t),α m (t), β m (t)}. x m (t) represents the task being computed locally or offloaded to an edge server. Specifically, when x m When (t) = 0, the task will be calculated locally. m (t)∈{1,2,...,N} means that the computation will be performed at the edge server n(n∈{1,2,...,N}). m (t)∈{1, 2, ..., C} is the transmission channel selected by device m. m (t)∈[0,1] defines the percentage of computing resources allocated to local computing. m (t)∈[0, 1] represents the percentage of transmission power during data transmission.

[0073] S200: Based on the action decision of the device user in the current state, the device user interacts with the environment to obtain the interaction delay and interaction energy consumption of the device user, and calculates them according to the completion status of the data interaction to obtain the delay reward and energy consumption reward;

[0074] Specifically, the device user interacts with the environment based on the action decision in the current state and performs the corresponding computing task; if the computing task is to perform local computing, the delay and energy consumption are calculated according to the computing resources of the user device and the task requirements, and the delay of the local computing and the energy consumption of the local computing are determined; if the computing task is to perform offloaded computing, the transmission delay of the offloaded computing, the edge server computing delay and the transmission energy consumption of the offloaded computing are determined according to the computing power of the transmission channel and the edge server; the interaction delay of the device user in the preset time interval is determined by combining the delay of the local computing, the transmission delay of the offloaded computing and the edge server computing delay; the interaction energy consumption of the device user in the preset time interval is determined by combining the energy consumption of the local computing and the transmission energy consumption of the offloaded computing; based on the completion of the data interaction, the interaction delay of the device user in the preset time interval and the interaction energy consumption of the device user in the preset time interval are rewarded and calculated respectively to obtain a delay reward and an energy consumption reward.

[0075] In this embodiment, for performing local calculations, when x m When (t) = 0, it means that the task generated by device m in time interval t is performed locally. We assume that each device uses a single CPU to perform task calculations, and the maximum local computing resources of each device are and maximum transmission power are different. We use α m (t)∈[0,1] and β m (t)∈[0,1] to define the local computation assigned to task G m The percentage of computing resources of (t) and the percentage of transmission power during data transmission. The delay of device m processing local tasks is:

[0076]

[0077] The corresponding energy consumption is:

[0078]

[0079] in It is the energy consumed by the device's CPU calculations per cycle, which depends on the actual architecture of the chip.

[0080] For performing offload calculations, when x mWhen (t)∈{1, 2, ..., N}, the task is offloaded. We divide offloaded computation into two parts: data transmission and edge server computation. Since the amount of returned data is much smaller than the amount of uploaded data, the latency associated with the return of the results is negligible. The total latency of offloaded computation primarily includes the data transmission latency and the computation latency at the edge server, while the energy consumption only includes the energy consumed by the device during data transmission.

[0081] We use the WINNER II path loss model to calculate path loss and the Rician channel model to calculate channel gain.

[0082] We use (x m ,y m ) and (x n ,y n ) represent the coordinate positions of device m and edge server n respectively. Accordingly, we can calculate the distance between the two using the Euclidean distance formula:

[0083] We use the WINNER II channel model to calculate the path loss. c The path loss expression for the case is as follows:

[0084]

[0085] in, is the reference path loss, A W ·log 10 (d) is the distance-related term, which has a logarithmic growth relationship with the propagation distance d. is a frequency-dependent term, which is related to the carrier frequency f c It shows a logarithmic growth relationship.

[0086] According to the Rice channel model, the channel gain can be calculated according to the following formula:

[0087]

[0088] in, is the variance of the multipath scattering component, κ is the Rice factor, θ is the random phase of the direct path, which is usually uniformly distributed in the range [0, 2π], CN(0, σ 2 ) represents the multipath scattering component of the complex Gaussian distribution.

[0089] The channel gain of device m can be obtained from the above formula, and the corresponding signal-to-noise ratio can be calculated:

[0090]

[0091] Where W is the channel bandwidth, N0 is the noise power spectral density, which is used to indicate the intensity of background noise. is the actual transmit power. Perform tasks on behalf of other devices m (t) Interference caused during task transmission.

[0092] From the above formula, we can get the upload data rate of device m during task offloading in time interval t:

[0093] r m =W c log2(1+SINR m )

[0094] The transmission delay of this task can be obtained by the following formula:

[0095]

[0096] The corresponding transmission energy consumption is:

[0097]

[0098] Furthermore, in order to balance tasks with less computational requirements and larger tasks and avoid wasting resources, in our system, 20% of the computing frequency of the edge server is evenly divided by the computing tasks offloaded to the edge server, and 80% of the computing frequency is allocated according to the proportion of computing cycles required by the tasks.

[0099] From this, we can get the computational delay of the edge server:

[0100]

[0101] in, represents the computation frequency of the task assigned to device m by edge server n in time interval t.

[0102] In summary, the latency of offloaded computing is the sum of the transmission latency and the edge server computing latency, that is:

[0103]

[0104] The energy consumption of offloading calculation only includes transmission energy consumption:

[0105]

[0106] Finally, our optimization goal is to minimize task execution latency and energy consumption. To perform targeted optimization more effectively, we calculate latency and reward separately.

[0107] Based on the above local computing model and offload computing model, we can get the latency expression of the task of device m in time interval t as:

[0108]

[0109] Among them, the indicator function When x m It is 1 when (t)=0, otherwise it is 0.

[0110] The corresponding energy consumption expression is:

[0111]

[0112] Therefore, the computation offloading problem is modeled as a problem:

[0113]

[0114] stC1:x m (t)∈0,1,…,N

[0115] C2:c m (t)∈1,...,C

[0116] C3:α m (t)∈[0,1]

[0117] Ct:β m (t)∈[0,1]

[0118] C5:d m (t)≤d m,max (t)

[0119] C1 limits the decision-making process for devices between local computing and offloaded computing. For offloaded computing, devices must select an edge server from a set of edge servers for task offload. C2 limits the channel selection of devices. C3 and Ct limit the range of computing resources allocated for local computing and the power range of data transmission, respectively. C5 limits the latency of task completion to less than the maximum allowable latency.

[0120] Furthermore, our goal is to obtain higher total utility and increase task completion rate for delay-sensitive multi-objective tasks. Due to the strong dynamics of MEC environments, traditional methods that require precise mathematical models and complex designs are not competent.

[0121] In order to solve the above problems, the embodiment of the present invention proposes an algorithm based on deep reinforcement learning, namely MC-PPO. Figure 3As shown in the figure, to enable each agent to make decisions independently based on local observations while also being able to collaborate with other agents, we first introduced the CTDE mechanism. Then, to enable different optimization strategies based on different training results, we introduced the multi-critic mechanism. The following describes the Markov decision process (MDP), the PPO algorithm, the CTDE mechanism, and the multi-critic mechanism.

[0122] The state space s of device m at time interval t m (t) Includes the attributes G of the relevant tasks m (t), and device related information. We use G m (t) = {s m (t), c m (t), d m，max (t)} to describe the task, where s m (t) represents the amount of task data, c m (t) represents the computational density, that is, the number of CPU cycles required for each bit-sized task, d m,max (t) represents the maximum allowed delay of the task. In addition, we believe that the information of the local device is also important for the decision-making of the agent, that is, the maximum computing resources of the device and maximum transmission power They are important factors in the decision-making and resource allocation of joint computing offloading. Therefore, the local state space of device m in time interval t is

[0123] Our model needs to make decisions by comprehensively considering computation offloading and resource allocation. Its action space can be expressed as a m (t) = {x m (t), ch m (t), α m (t), β m (t)}. x m (t) represents the task being computed locally or on which edge server the computation is offloaded. m (t) is the transmission channel selected by the agent. m (t) is the computing resource allocated for local computing. m (t) is the transmission power during data transmission.

[0124] In this embodiment of the present invention, we no longer directly weight the latency reward and energy consumption reward as the total utility. We calculate the latency and energy consumption rewards separately, specifically, as shown in the following formula:

[0125]

[0126] In the above formula, represents the delayed reward, represents energy consumption reward, d m (t) represents the interaction delay between the device user and the preset time interval, e m (t) represents the interaction energy consumption of the device user in the preset time interval. When the task is completed on time, the delay reward and energy consumption are respectively m (t) and energy consumption e m Negative value of (t), p d and p e are the timeout penalty constants representing delay and energy consumption respectively.

[0127] When a task is completed within the maximum allowed experiment, the rewards for latency and energy consumption are the negative values of the actual latency and energy consumption, respectively. When a task times out and is discarded, a constant much larger than the latency or energy consumption will be used as a penalty.

[0128] Therefore, each device interacts with the environment based on the selected action and performs the corresponding task processing. For local computing tasks, the latency and energy consumption are calculated based on the device's computing resources and task requirements. For offloaded computing tasks, the transmission latency, edge server computing latency, and transmission energy consumption are calculated based on the transmission channel, the edge server's computing power, and other factors. Based on the actual completion of the task, a latency reward and energy consumption reward are calculated. If the task is completed within the maximum allowable latency, the latency reward is the negative of the actual latency value, and the energy consumption reward is the negative of the actual energy consumption value. If the task times out and is discarded, the latency reward and energy consumption reward are respectively set to preset large constant values.

[0129] S300: Based on the centralized training distributed execution mechanism and the multi-critic mechanism, combined with the delay reward and the energy consumption reward, the device user's action decision is trained and updated to obtain the trained device user;

[0130] Specifically, the delay reward and energy consumption reward are fed back to the user device to obtain the new state of the user device; the current state, action decision, delay reward, energy consumption reward and new state of the user device are combined to build an experience pool and store it in the trajectory collector; based on the multi-critic mechanism, a critic is assigned to the delay reward and energy consumption reward, namely the delay critic network and the energy consumption critic network, and the loss function of the delay critic network and the loss function of the energy consumption critic network are determined; the data in the trajectory collector is extracted, the delay critic network estimates the delay value function according to the delay reward and the new state, and the energy consumption critic network estimates the energy consumption value function according to the energy consumption reward and the new state, and the loss function of the delay critic network and the loss function of the energy consumption critic network are minimized to obtain the delay optimization effect and the energy consumption optimization effect; if the delay optimization effect is less than the energy consumption optimization effect, the policy network is updated through the delay value function, and if the energy consumption optimization effect is less than the delay optimization effect, the policy network is updated through the energy consumption value function, and the action decision of the device user is trained and updated to obtain the trained device.

[0131] In this embodiment, PPO is an improvement on TRPO (Trust Region Policy Optimization). Both are reinforcement learning algorithms based on policy gradients, and both ensure the stability of policy updates by limiting the difference between the new and old policies. TRPO uses KL divergence (Kullback-Leibler divergence) to constrain the difference between the new and old policies, ensuring that the update step is performed within the neighborhood of the current policy. Because TRPO needs to use Taylor expansion approximation, conjugate gradient, linear search and other methods to solve directly, the calculation is relatively expensive. PPO simplifies the training process by clipping the objective function, which is expressed as:

[0132]

[0133] Among them, cli[(x, l, r) = max(min(x, r), l), ∈ is a hyperparameter, indicating the range of truncation (clip).

[0134] Advantage function A π (t) is calculated using the Generalized Advantage Estimation (GAE), which is expressed as:

[0135]

[0136] Therefore, the loss function of the actor network for PPO is:

[0137]

[0138] The update function for the critic network in PPO is:

[0139]

[0140] In the centralized training phase, global information is used to improve learning efficiency and effectiveness, while in the distributed execution phase, each agent relies only on its own local information and strategies to make decisions, thereby achieving decentralized operations.

[0141] In MAPPO, the centralized state value function of device m can be updated according to the following loss function, which is expressed as:

[0142]

[0143] Among them S t =concat(s m,t ), It is the concatenation of the state sets of all devices in the time interval t. The state value function will be replaced by the central state value function, which will affect the generalized advantage estimation and finally affect the update of the actor network. Its expression is:

[0144]

[0145] In previous studies, the delay and energy consumption information fed back by the environment are usually weighted and processed as rewards to train the model, that is, r = μ d r d (t)+μ e r e (t), where μ d and μ e are the weights for latency rewards and energy consumption rewards, respectively. However, these methods cause the actor network to use the same optimization strategy when different latency and energy consumption combinations have the same utility value, resulting in the inability to perform targeted optimization. To address this issue, we introduced a multi-critic mechanism, assigning a critic to latency and energy consumption respectively, corresponding to different optimization objectives. Specifically, the critic network for device m has:

[0146]

[0147] In the above formula, represents the loss function of the delayed critic network, represents the loss function of the energy-consumption critic network, S t Represents the state of all agents in the environment at the current moment, S t+1 Represents the state of all agents in the environment at the next moment, and They represent delay rewards and energy consumption rewards respectively, γ represents the discount factor. The closer it is to 1, the more attention is paid to long-term cumulative rewards, and the closer it is to 0, the more consideration is given to short-term rewards. d (·) and V e (·) denotes the delay critic network and energy consumption critic network, respectively.

[0148] For the actor network of device m, the advantage function is calculated as follows:

[0149]

[0150] In the above formula, represents the delay value function, represents the energy consumption value function, γ represents the discount factor, and λ represents a hyperparameter used to calculate the generalized advantage estimate (GAE) to balance the bias and variance of the advantage function. They represent the delay reward and energy consumption reward obtained by the agent at time t+l, S t+l+1 and S t+l denotes the state of all agents in the environment at time t+l+1 and time t+l, respectively. l denotes the index variable in the generalized advantage estimate, which is used to traverse subsequent time steps starting from the current time step t. It indicates the number of TD residual steps considered when computing the advantage estimate.

[0151] When the delay reward obtained after the action selected by the strategy interacts with the environment is less than the energy consumption reward, it means that the strategy in the current training stage is poorly optimized for delay. We choose the delay GAE to update the actor network strategy. Otherwise, we use the energy consumption GAE. Specifically, the relationship between delay and energy consumption is:

[0152]

[0153] Therefore, the delay critic network estimates the delay value function based on the delay reward and the new state, and the energy consumption critic network estimates the energy consumption value function based on the energy consumption reward and the new state. According to the optimization effect of delay and energy consumption under the current strategy, the advantage function of the corresponding critic network is selected to update the policy network. For example, if the delay optimization effect is poor under the current strategy, the delay advantage function is used to update the policy network first, and vice versa. Determine whether the effect obtained by the algorithm meets the convergence conditions. If the convergence conditions are not met, return to step S100 to continue execution; if the convergence conditions are met, save the final policy network parameters, end the training process, and deploy the trained agent to the corresponding device, so that the device can make decisions independently based on local observation results.

[0154] S400: Making independent decisions based on the trained device user to execute the computing offloading task.

[0155] In summary, the embodiment of the present invention consists of three parts: an actor network and two critic networks corresponding to latency and energy consumption, respectively. The actor network is a two-layer neural network. The hidden layer contains 128 hidden nodes. The output layer outputs the mean and standard deviation of a normal distribution. Action decisions are obtained by sampling the normal distribution. The hidden layer uses the ReLU activation function. The critic network is also a two-layer neural network. The hidden layer contains 128 hidden nodes and uses the ReLU activation function. The learning rate of the actor network is 10 -t , the learning rate of the critic network is 5×10 -3 , the discount factor γ is 0.9, the generalized advantage estimate λ is 0.9, and the cutoff parameter ∈ for PPO is 0.2.

[0156] The pseudo code for its specific implementation is as follows:

[0157]

[0158] Finally, a simulation experiment was conducted on the embodiment of the present invention. The environment was within a rectangular range of 150 meters to 150 meters, containing 20 to 110 devices, 5 to 15 edge servers, and 5 to 15 channels. The model consists of three parts, including an actor network and two critic networks corresponding to latency and energy consumption respectively. The actor network is a two-layer neural network. The hidden layer contains 128 hidden nodes. The output layer outputs the mean and standard of the normal distribution. The action decision is obtained by sampling the normal distribution. The hidden layer uses the ReLU activation function. The critic network is also a two-layer neural network. The hidden layer contains 128 hidden nodes. The ReLU activation function is used. The learning rate of the actor network is 10 -t , the learning rate of the critic network is 5×10 -3 , the discount factor γ is 0.9, the generalized advantage estimate λ is 0.9, each training round is divided into 10 time slots, and the PPO cutoff value ∈ is 0.2. Unless otherwise specified or changed, our default experimental environment is: 40 devices, 7 edge servers, and 9 channels. The experiment compares three other common reinforcement learning algorithms and a random algorithm: 1) SAC-based algorithm; 2) DDPG-based algorithm; 3) A2C-based algorithm; 4) random offloading algorithm (RO). The total utility calculation formula is Other experimental environment parameters can be found in Table 1.

[0159] Table 1 Simulation experiment parameter settings

[0160]

[0161] Further, for the task offloading and resource allocation in different device number scenarios of heterogeneous devices, the number of devices ranges from 20 to 110 devices, there are 7 edge servers and 9 channels. The experimental results are as follows: Figure 4 shown.

[0162] 60 devices, 7 edge servers, 9 channels. MC-PPO converges after 500 iterations, while DDPG, SAC, and A2C converge after 900, 500, and 800 iterations, respectively. Our proposed model converges faster and achieves higher total utility. The experimental results are as follows: Figure 5 As shown in Table 2.

[0163] Table 2 Total utility data of different numbers of equipment

[0164] Total utility 20 30 40 50 60 70 80 90 100 110 MC-PPO -0.854 -0.940 -1.135 -1.014 -1.064 -1.062 -1.060 -1.066 -1.043 -1.058 DDPG -5.178 -5.589 -5.270 -4.402 -3.563 -4.441 -3.590 -4.790 -5.539 -4.653 SAC -3.429 -2.981 -2.795 -3.609 -3.072 -3.322 -2.467 -2.749 -2.621 -2.915 A2C -1.987 -2.014 -1.719 -1.533 -1.572 -1.574 -1.439 -1.597 -1.463 -1.555 RO -5.668 -6.379 -7.441 -7.930 -8.823 -9.393 -9.465 -9.359 -9.738 -10.108

[0165] Therefore, compared with the prior art, the improvements of the embodiments of the present invention are:

[0166] 1) Design independent critic networks V for latency and energy consumption d and V e , dynamically select the advantage function to update the strategy. According to the real-time reward comparison and Prioritizing latency or energy consumption avoids the policy confusion caused by traditional weighted summation. By separating optimization objectives, precise multi-objective dynamic optimization is achieved. Experiments have shown significant improvements in latency, energy consumption, and overall task completion rates.

[0167] 2) Centralized Training Distributed Execution (CTDE) mechanism: During the training phase, the cloud computing center leverages global information (device status, channel interference, and server load) to optimize the policy network and enhance multi-device collaboration. During the execution phase, devices make independent decisions based solely on local state (task attributes and local resources), ensuring flexibility and scalability. This allows agents to share information during the training phase to facilitate coordination. Furthermore, distributed execution offers excellent scalability, adapting to the challenges posed by environments with varying numbers of devices.

[0168] Reference Figure 2 , a deep reinforcement learning computation offloading system based on a multi-critic mechanism, including:

[0169] The first module 201 is used to obtain the local state vector of the device user and make a decision through a multi-critic proximal strategy optimization algorithm to obtain an action decision in the current state of the device user;

[0170] The second module 202 is used to interact with the environment based on the action decision of the device user in the current state, obtain the interaction delay and interaction energy consumption of the device user, and calculate them according to the completion status of the data interaction to obtain the delay reward and energy consumption reward;

[0171] The third module 203 is used to train and update the action decision of the device user based on the centralized training distributed execution mechanism and the multi-critic mechanism, combined with the delay reward and the energy consumption reward, to obtain a trained device user;

[0172] The fourth module 204 is configured to make independent decisions based on the trained device user to perform computation offloading tasks.

[0173] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A deep reinforcement learning computation offloading method based on a multi-critic mechanism, characterized by: The following steps are involved: Obtain the local state vector of the device user and make a decision through the multi-critic proximal policy optimization algorithm to obtain the action decision of the device user in the current state; Based on the action decisions of the device user in the current state, the device user interacts with the environment to obtain data, obtain the interaction delay and interaction energy consumption of the device user, and calculate them according to the completion status of the data interaction to obtain delay rewards and energy consumption rewards; Based on the centralized training distributed execution mechanism and multi-critic mechanism, combined with delay rewards and energy consumption rewards, the device user's action decision is trained and updated to obtain the trained device user; Make independent decisions based on trained device users to perform computational offloading tasks.

2. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 1 is characterized in that: The step of obtaining the local state vector of the device user and making a decision through the multi-critic proximal strategy optimization algorithm to obtain the action decision of the device user in the current state specifically includes: Obtain the task information and status information of the device user in the current time interval and construct a local state vector; Based on the observation information of the environment, the local state vector is forward propagated through the actor network model to obtain the mean and standard deviation of the Gaussian distribution; The mean and standard deviation values of the Gaussian distribution are sampled to determine the action decision under the current state of the device user.

3. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 2 is characterized in that: The specific expressions of the action decisions of the multiple device users in their current states are as follows: a m (t)={x m (t),ch m (t),α m (t),β m (t)} In the above formula, a m (t) represents the action decision in the current state, x m (t) indicates whether the task is computed locally or on which edge server the computation is offloaded. m (t) represents the transmission channel selected by the agent, α m (t) represents the computing resources allocated for local computing, β m (t) represents the transmission power during data transmission.

4. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 3 is characterized in that: The step of interacting with the environment based on the action decision of the device user in the current state, obtaining the interaction delay and interaction energy consumption of the device user, and calculating the delay reward and energy consumption reward based on the completion of the data interaction specifically includes: Device users interact with the environment based on their current state and perform corresponding computing tasks. If the computing task is to perform local computing, the latency and energy consumption are calculated based on the computing resources of the user device and the task requirements, and the latency and energy consumption of the local computing are determined; If the computing task is to perform offloaded computing, determine the transmission delay of the offloaded computing, the edge server computing delay, and the transmission energy consumption of the offloaded computing based on the transmission channel and the computing capacity of the edge server; Determine the device user interaction latency during a preset time interval by combining the local computation latency, the offloaded computation transmission latency, and the edge server computation latency. Combine the energy consumption of local computing with the transmission energy consumption of offloaded computing to determine the interaction energy consumption of device users during preset time intervals; According to the completion of data interaction, the interaction delay of the device user in the preset time interval and the interaction energy consumption of the device user in the preset time interval are rewarded and calculated respectively to obtain the delay reward and energy consumption reward.

5. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 4 is characterized in that: The expression of the interaction delay of the device user in the preset time interval is specifically as follows: In the above formula, d m (t) represents the interaction delay between the device user and the preset time interval, Indicates the latency of local computing, represents the sum of the transmission delay of the offloaded calculation and the edge server calculation delay, x m (t) represents the indicator function; The expression of the interaction energy consumption of the device user in the preset time interval is specifically as follows: In the above formula, e m (t) represents the interaction energy consumption of the device user in the preset time interval, represents the energy consumption of local computing, represents the transmission energy consumption of offloaded calculation, x m (t) represents the indicator function.

6. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 5 is characterized in that: The calculation expressions of the delay reward and energy consumption reward are as follows: In the above formula, represents the delayed reward, represents energy consumption reward, d m (t) represents the interaction delay between the device user and the preset time interval, e m (t) represents the interaction energy consumption of the device user in the preset time interval, p d and p e are the timeout penalty constants representing delay and energy consumption respectively.

7. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 6 is characterized in that: The step of training and updating the action decision of the device user based on the centralized training distributed execution mechanism and the multi-critic mechanism, combined with the delay reward and the energy consumption reward, to obtain the trained device user specifically includes: Feedback the latency reward and energy consumption reward to the user device to obtain the new status of the user device; Combining the current state of the user device, action decision, delay reward, energy consumption reward and new state, an experience pool is constructed and stored in the trajectory collector; Based on the multi-critic mechanism, a critic is assigned to each of the delay reward and the energy consumption reward, namely the delay critic network and the energy consumption critic network, and the loss function of the delay critic network and the loss function of the energy consumption critic network are determined; Extract data from the trajectory collector. The delay critic network estimates the delay value function based on the delay reward and the new state. The energy critic network estimates the energy value function based on the energy reward and the new state. Minimize the loss function of the delay critic network and the loss function of the energy critic network to obtain the delay optimization effect and energy consumption optimization effect. If the delay optimization effect is less than the energy consumption optimization effect, the policy network is updated through the delay value function. If the energy consumption optimization effect is less than the delay optimization effect, the policy network is updated through the energy consumption value function. The action decision of the device user is trained and updated to obtain the trained device.

8. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 7 is characterized in that: The loss function expressions of the delay critic network and the energy consumption critic network are specifically as follows: In the above formula, represents the loss function of the delayed critic network, represents the loss function of the energy-consumption critic network, S t Represents the state of all agents in the environment at the current moment, S t+1 Represents the state of all agents in the environment at the next moment, and Represent delay reward and energy consumption reward respectively, γ represents discount factor, V d (·) and V e (·) denotes the delay critic network and energy consumption critic network, respectively.

9. The deep reinforcement learning computation offloading method based on the multi-critic mechanism according to claim 8 is characterized in that: The expressions of the delay value function and the energy consumption value function are specifically as follows: In the above formula, represents the delay value function, represents the energy consumption value function, γ represents the discount factor, and λ represents a hyperparameter used to calculate the generalized advantage estimate (GAE). They represent the delay reward and energy consumption reward obtained by the agent at time t+l, S t+l+1 and S t+l They represent the states of all agents in the environment at time t+l+1 and time t+l respectively, and l represents the index variable in the generalized advantage estimation.

10. A deep reinforcement learning computation offloading system based on a multi-critic mechanism, characterized by: Includes the following modules: The first module is used to obtain the local state vector of the device user and make a decision through the multi-critic proximal policy optimization algorithm to obtain the action decision of the device user in the current state; The second module is used to interact with the environment based on the device user's current action decisions, obtain the device user's interaction delay and interaction energy consumption, and calculate them based on the completion of the data interaction to obtain delay rewards and energy consumption rewards; The third module is used to train and update the device user's action decision based on the centralized training distributed execution mechanism and the multi-critic mechanism, combined with the delay reward and energy consumption reward, to obtain the trained device user; The fourth module is used to make independent decisions based on trained device users to perform computational offloading tasks.

Images