Method and device for offloading secure reinforcement learning tasks based on penalty mechanism

Abstract

The present disclosure relates to a method and device for unloading a secure reinforcement learning task based on a penalty mechanism, the method comprising: receiving a computing service request from a user end; obtaining current status information of the user end; inputting the status information into a secure reinforcement learning model to obtain a decision result for the computing service request, wherein the secure reinforcement learning model is obtained by training an objective function that maximizes an obstacle function, the obstacle function is used to punitively adjust the objective function value according to a first constraint condition during the training process of the secure reinforcement learning model, the decision result is to determine to which of multiple servers the task corresponding to the computing service request is unloaded for processing, the multiple servers including a cloud server and at least one edge server; the task corresponding to the computing service request is unloaded based on the decision result. The obstacle function can automatically trigger a pre-punishment mechanism to effectively avoid the occurrence of violations.

Description

Technical Field

[0001] The present disclosure relates to the field of task offloading, and in particular to a method, device, electronic device, storage medium, system and computer program product for secure reinforcement learning task offloading based on a penalty mechanism. Background Art

[0002] With the rise of new generation networks and the popularity of mobile devices, the amount of data transmission and the diverse computing needs are growing exponentially. In this context, the emergence of new computing-intensive applications has put forward higher requirements on the computing power of devices, while traditional local computing often fails to meet these requirements. In order to meet this challenge, solve a series of delay problems brought by cloud computing, and make full use of computing resources at the edge of the network, many studies have been conducted on the mobile edge computing paradigm. Mobile edge computing offloads local computing tasks to servers at the edge of the network or cloud servers, thus achieving flexible scheduling and efficient utilization of computing resources, thereby significantly reducing the delay of task processing and improving the computing efficiency of the overall system.

[0003] Safety reinforcement learning can solve the multi-constrained task offloading problem. Usually, the Lagrangian method is introduced in reinforcement learning to solve it, which is a post-remediation method and may cause system performance fluctuations and parameter overshoot. Summary of the invention

[0004] The penalty-based secure reinforcement learning task offloading method, device, electronic device, storage medium, system, and computer program product provided by the exemplary embodiments of the present disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.

[0005] According to one aspect of the present disclosure, a method for unloading a secure reinforcement learning task based on a penalty mechanism is provided, the method comprising: receiving a computing service request from a user terminal; obtaining current status information of the user terminal; inputting the status information into a secure reinforcement learning model to obtain a decision result for the computing service request, wherein the secure reinforcement learning model is trained by maximizing an objective function including an obstacle function, and the obstacle function is used to perform a penalty adjustment on the objective function value according to a first constraint during the training of the secure reinforcement learning model, the first constraint being a constraint on at least one of energy consumption, delay, and bandwidth, the decision result being to determine to which of a plurality of servers the task corresponding to the computing service request is to be unloaded for processing, the plurality of servers comprising a cloud server and at least one edge server; and the task corresponding to the computing service request is unloaded based on the decision result.

[0006] Optionally, the barrier function is used to perform a punitive adjustment on the objective function value through a larger penalty term when the execution result of the current strategy of the security reinforcement learning model is closer to the first constraint condition.

[0007] Optionally, during the training process of the security reinforcement learning model, the degree of difference between the strategy of the previous preset stage and the hybrid strategy of the security reinforcement learning model is also limited based on a second constraint condition, wherein the second constraint condition is a trust domain constraint, and the hybrid strategy is obtained by linearly weighting the current strategy of the security reinforcement learning model and the strategy of the previous preset stage.

[0008] Optionally, the degree of difference between the strategy of the previous preset stage and the hybrid strategy is obtained by calculating KL divergence.

[0009] Optionally, the current strategy and the strategy of the previous preset stage are both Gaussian distributions, and the mixed strategy is a Gaussian mixture distribution.

[0010] Optionally, the step of calculating the degree of difference between the strategy of the previous preset stage and the hybrid strategy includes: determining the corresponding variational distribution based on the Gaussian mixture distribution state of the hybrid strategy, wherein the parameters of the variational distribution are optimizable; using the Jensen inequality to perform a lower bound estimate on the KL divergence between the hybrid strategy and the variational distribution to obtain the optimal solution for the parameters of the variational distribution; calculating the KL divergence between the strategy of the previous preset stage and the optimal solution for the parameters of the variational distribution to obtain the degree of difference between the strategy of the previous preset stage and the hybrid strategy.

[0011] According to another aspect of the present disclosure, a device for unloading secure reinforcement learning tasks based on a penalty mechanism is also provided, the device comprising: a receiving unit, configured to receive a computing service request from a user terminal; an acquiring unit, configured to acquire current status information of the user terminal; a decision unit, configured to input the status information into a secure reinforcement learning model to obtain a decision result for the computing service request, wherein the secure reinforcement learning model is trained by maximizing an objective function including an obstacle function, and the obstacle function is used to perform a punitive adjustment on the objective function value according to a first constraint during the training of the secure reinforcement learning model, the first constraint being a constraint on at least one of energy consumption, delay, and bandwidth, the decision result being to determine to which of a plurality of servers the task corresponding to the computing service request is to be unloaded for processing, the plurality of servers comprising a cloud server or an edge server among a plurality of edge servers; the unloading unit, configured to unload the task corresponding to the computing service request based on the decision result.

[0012] Optionally, the barrier function is used to perform a punitive adjustment on the objective function value through a larger penalty term when the execution result of the current strategy of the security reinforcement learning model is closer to the first constraint condition.

[0013] Optionally, during the training process of the security reinforcement learning model, the security reinforcement learning model also limits the degree of difference between the strategy of a previous preset stage and the hybrid strategy of the security reinforcement learning model based on a second constraint condition, wherein the second constraint condition is a trust domain constraint, and the hybrid strategy is obtained by linearly weighting the current strategy of the security reinforcement learning model and the strategy of a previous preset stage.

[0014] Optionally, the degree of difference between the strategy of the previous preset stage and the hybrid strategy is obtained by calculating KL divergence.

[0015] Optionally, the current strategy and the strategy of the previous preset stage are both Gaussian distributions, and the mixed strategy is a Gaussian mixture distribution.

[0016] Optionally, the step of calculating the degree of difference between the strategy of the previous preset stage and the hybrid strategy includes: determining the corresponding variational distribution based on the Gaussian mixture distribution state of the hybrid strategy, wherein the parameters of the variational distribution are optimizable; using the Jensen inequality to perform a lower bound estimate on the KL divergence between the hybrid strategy and the variational distribution to obtain the optimal solution for the parameters of the variational distribution; calculating the KL divergence between the strategy of the previous preset stage and the optimal solution for the parameters of the variational distribution to obtain the degree of difference between the strategy of the previous preset stage and the hybrid strategy.

[0017] According to another aspect of an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the penalty mechanism-based secure reinforcement learning task offloading methods described above.

[0018] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the penalty mechanism-based secure reinforcement learning task offloading methods described above.

[0019] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the penalty mechanism-based secure reinforcement learning task offloading methods described above.

[0020] According to another aspect of an embodiment of the present disclosure, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the penalty mechanism-based secure reinforcement learning task offloading method as described in any one of the above.

[0021] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0022] According to the disclosed safe reinforcement learning task offloading method, device, electronic device, storage medium, system and computer program product based on penalty mechanism, the concept of obstacle function is introduced and integrated into the pre-punishment mechanism. The core of this mechanism is to build an early warning system that can perceive and respond to the approach of constraint boundaries in advance by integrating obstacle terms into the objective function. When the strategy gradually approaches the constraint boundary during the optimization process, the obstacle function can automatically trigger the pre-punishment mechanism and make forward-looking punitive adjustments to potential violations, thereby effectively avoiding the occurrence of violations.

[0023] In addition, through the penalty function, the system can evaluate the distance between the current solution and the constraint boundary in real time, and impose appropriate penalties based on the evaluation results to ensure that the solution always remains within the feasible domain or approaches it.

[0024] In addition, complex policy spaces can be simulated and approximated by constructing hybrid policies.

[0025] In addition, by using the secure reinforcement learning method to introduce the KL (Kullback-Leibler) divergence constraint, the optimization of the current strategy can be controlled so that the difference between the current strategy and the previous strategy is not too large.

[0026] In addition, by using variational approximation technology and deriving the optimal solution of the variational parameters through the Jensen inequality, we can more accurately calculate and give a lower bound estimate of the KL divergence under the Gaussian mixture model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0028] Figure 1 A schematic diagram showing a mobile edge computing system in an exemplary embodiment of the present disclosure;

[0029] Figure 2 A flowchart showing a method for offloading a secure reinforcement learning task based on a penalty mechanism in an exemplary embodiment of the present disclosure is shown;

[0030] Figure 3A flowchart of a method for offloading a secure reinforcement learning task based on a penalty mechanism in an exemplary embodiment of the present disclosure is shown;

[0031] Figure 4 A block diagram showing a device for offloading secure reinforcement learning tasks based on a penalty mechanism in an exemplary embodiment of the present disclosure is shown;

[0032] Figure 5 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0035] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0036] In mobile edge computing systems, task offloading mechanisms are the key to achieving low-latency and energy-efficient computing services. However, task offloading in mobile edge networks often faces major challenges such as improper resource allocation and network latency.

[0037] In practical applications, although existing optimization methods can improve system performance to a certain extent, they also face major problems such as constraint violation and instability. This is mainly because these existing optimization methods often fail to timely consider specific constraints that must be followed in system operation (such as resource limitations, etc.) while pursuing the optimal solution.

[0038] For example, in the optimization of safe reinforcement learning policies, the Lagrangian method is widely used because it is simple and effective in achieving optimal constraint satisfaction. This method introduces Lagrangian multipliers to transform constraints into part of the objective function, thereby indirectly considering the impact of constraints in the optimization process.

[0039] However, the Lagrangian method also has obvious limitations in the safe reinforcement learning training process, which hinders its application in the field of constrained optimization.

[0040] On the one hand, due to the hysteresis of Lagrange multiplier control, it is difficult to respond to changes in the system state in a timely manner; on the other hand, when the strategy approaches the constraint boundary, the lack of appropriate penalties and gradient changes may cause oscillation and overshoot in the system performance, thereby increasing the difficulty of ensuring safety.

[0041] In order to achieve the optimization goal while ensuring system security, it is necessary to implement constrained policy optimization to seek the best strategy while ensuring compliance with specific constraints.

[0042] In order to solve the above problems, the present disclosure provides a method, device, electronic device, storage medium, system and computer program product for safe reinforcement learning task offloading based on a penalty mechanism, introduces the concept of obstacle function, and integrates it into the pre-punishment mechanism. The core of this mechanism is to integrate the obstacle term into the objective function to build an early warning system that can perceive and respond to the approach of the constraint boundary in advance. When the strategy gradually approaches the constraint boundary during the optimization process, the obstacle function can automatically trigger the pre-punishment mechanism and make forward-looking punitive adjustments to potential violations, thereby effectively avoiding the occurrence of violations.

[0043] Next, we will refer to Figures 1 to 5 The present invention describes a penalty-based secure reinforcement learning task offloading method, device, electronic device, storage medium, system, and computer program product.

[0044] The present disclosure constructs a task offloading system model (security reinforcement learning model) in a mobile edge network, which integrates the interaction mechanism between the server and the user in the mobile edge computing system.

[0045] Figure 1 A schematic diagram of a mobile edge computing system in an exemplary embodiment of the present disclosure is shown.

[0046] Reference Figure 1In the mobile edge computing system, the server side can be composed of a powerful remote cloud server and several widely distributed edge servers. The cloud server contains a large storage, based on which, the cloud server can be used for computing services such as big data storage, big data analysis, and other computing services. The main data transmission path between the cloud server and the edge server is the backbone connection. In the edge server, the path for data transmission between the user end or other servers is a channel, such as Ethernet, optical channel, etc.; the router can be used to manage the network connection of multiple edge devices to realize communication and data transmission between devices; the base station can provide wireless signal coverage and data transmission for mobile devices; the storage can be used to store large amounts of data, as well as support real-time processing and analysis of these data; the devices in the edge server can be connected through nodes.

[0047] The user side covers multiple mobile users with different computing needs, and their task requests constitute the main objects processed by the mobile edge computing system. These servers on the server side can work together to provide high-quality computing services for various industries and types of user sides, for example, providing computing services for user sides related to healthcare, virtual reality / augmented reality, economic autonomous driving, robots, drones, the Internet of Things, industry, etc.

[0048] Figure 2 A flowchart of a method for offloading a secure reinforcement learning task based on a penalty mechanism in an exemplary embodiment of the present disclosure is shown.

[0049] Reference Figure 2 , in step 201, a computing service request from a user is received.

[0050] Specifically, the task offloading system model can be designed as a continuous time system, but in actual operation, the decision steps can be discretized and defined as multiple time slots. At the beginning of each time slot, the task offloading system can receive mobile edge computing service requests from multiple different users on the user side. For each user, this series of computing service requests exists in the form of tasks, and the arrival time of these tasks can be regarded as randomly arriving at the server following a Poisson distribution.

[0051] It can be assumed that the task arrival rate of each user follows the parameter The Poisson distribution of the task arrival time is uncertain, but it obeys statistical laws as a whole. In order to manage these tasks more effectively, a first-in-first-out queue strategy can be adopted to put the arriving tasks into the queue one by one according to their arrival order and wait for the system to process them. At the end of each time slot, the system can check the first task in the queue and choose to assign the first task in the queue to the remote cloud server or edge server for processing based on the current system status.

[0052] This decision process needs to take into account multiple factors to ensure that the task can achieve the lowest energy consumption while meeting the latency requirements. When the decision is made, the system can immediately perform the offload operation, sending the first task in the queue to the server selected by the system and then removing it from the queue.

[0053] Since the wireless communication environment is very complex and considering the dynamic changes in signal strength, the Rayleigh fading channel model can be used to simulate this realistic scenario. Based on the Rayleigh fading channel model, a variant of the Shannon formula can be further introduced to quantify the data transmission rate between the user equipment (user end) and the server, so that the data transmission rate not only depends on the channel conditions, but is also proportional to the transmission power and the bandwidth allocated by the system, and inversely proportional to the noise power.

[0054] For the task offloading (system) model, the Markov decision process can be used to implement it, and in the mobile edge computing system environment, the corresponding state information, action information, reward function, state transition probability, etc. can be used through the policy optimization algorithm of the conditional Markov decision process to iteratively update the policy parameters to solve the Markov decision process.

[0055] In step 202, the current status information of the user terminal is obtained.

[0056] Specifically, the state information may include but is not limited to the task load of the current user end, the usage of computing resources, etc. The usage of computing resources may include but is not limited to the current load of the cloud service or edge server, network conditions, task queue length, etc.; the action information may be to offload the task to a cloud server or an edge server for execution; the reward function may be a function related to service delay, for example, it may include transmission delay and execution delay, etc.; the state transition probability can be used to describe how the state of the intelligent agent changes after performing a certain action according to factors such as task completion, arrival of new tasks, and changes in network conditions.

[0057] For the construction of a secure reinforcement learning model, we can first construct a constrained optimization problem based on the low latency requirements that task offloading in the mobile edge computing system must meet, consider task offloading as the strategy of the agent in the conditional Markov decision process, and parameterize the strategy space. The agent is in the environment of the mobile edge computing system, and can extract samples from the environment to update the strategy to gradually approach the approximate solution of the constrained optimization problem, thereby maximizing the system performance while ensuring that the constraints are met. It is understandable that the number of agents can be one or more.

[0058] Therefore, according to the exemplary embodiment of the present disclosure, in the strategy πThe constrained optimization problem of task offloading in mobile edge computing systems can be shown as follows:

[0059]

[0060] in, For users, For the server, is the uninstall decision (strategy) variable, representing the user The generated tasks are offloaded to the server , , which can clearly indicate in binary form whether a specific task should be offloaded from the user device to a server. Constraint 1 The task offloading variable must be binary. Ensure that each task can only be offloaded to one server; Constraint 3 Describes the energy constraints of the user. Discount factor Describes the discount of future goals, expectations Ability to consider the distribution of task sizes, user arrivals, and random strategies. It is a function related to the user's task load, CPU processing capability, channel transmission conditions, etc., and can be regarded as a negative correlation with the advantage function. To minimize network latency.

[0061] According to an exemplary embodiment of the present disclosure, the current strategy and the strategy of the previous preset stage are both Gaussian distributions, and the mixed strategy is a Gaussian mixed distribution.

[0062] Specifically, in order to adapt the strategy to the complex and changing mobile edge network environment, the Gaussian mixture model (GMM) can be used to construct a hybrid strategy to simulate and approximate the complex strategy space. The Gaussian mixture model can provide a solid framework for modeling complex distributions, and the hybrid strategy constructed using the Gaussian mixture model can be used to enhance the exploration capability of the secure reinforcement learning model.

[0063] During the training of the task offloading model, for the current strategy to be optimized , which can be a Gaussian distribution, this strategy can be Compared with the previous k The strategy for the training phase (i.e., the strategies of the previous preset stage) are linearly weighted mixed to obtain a mixed strategy in the form of a Gaussian mixture distribution (for example, the mixed strategy can be , in another example, the mixed strategy can be expressed as ,in, is a constant) as part of the second constraint.

[0064] According to an exemplary embodiment of the present disclosure, during the training process of the security reinforcement learning model, the degree of difference between the strategy of the previous preset stage and the hybrid strategy of the security reinforcement learning model is also limited based on the second constraint condition, the second constraint condition is the trust domain constraint, and the hybrid strategy is obtained by linearly weighting the current strategy of the security reinforcement learning model and the strategy of the previous preset stage.

[0065] Specifically, by limiting the difference between the strategy of the previous preset stage and the hybrid strategy through the second constraint condition, the optimization of the current strategy can be controlled not to differ too much from the strategy of the previous preset stage (that is, the optimization step should not be too large) during the training process of the task offloading model, so as to gradually optimize the strategy under the premise of smooth strategy changes. In addition, using the hybrid strategy framework to guide the resource allocation and scheduling process can provide a robust solution for task offloading and avoid system oscillations. At the same time, formulating the problem through the Gaussian mixture model can not only simplify the complexity of the problem, but also retain the key features of the original problem.

[0066] According to an exemplary embodiment of the present disclosure, when measuring the degree of difference between the strategy of the previous preset stage and the hybrid strategy, KL (Kullback-Leibler) divergence may be used for calculation.

[0067] When the KL divergence is used to calculate the degree of difference between the strategy of the previous preset stage and the mixed strategy, the variational approximation technique can be used to obtain a more accurate lower bound estimate of the KL divergence under the Gaussian mixture model.

[0068] According to an exemplary embodiment of the present disclosure, the corresponding variational distribution can be determined based on the Gaussian mixture distribution state of the hybrid strategy, wherein the parameters of the variational distribution are optimizable; the KL divergence between the hybrid strategy and the variational distribution is estimated by lower bound using the Jensen inequality to obtain the optimal solution for the parameters of the variational distribution; the KL divergence between the strategy of the previous preset stage and the optimal solution for the parameters of the variational distribution is calculated to obtain the degree of difference between the strategy of the previous preset stage and the hybrid strategy.

[0069] Specifically, in order to avoid directly calculating the difficult-to-solve posterior distribution (Gaussian mixture distribution), first, a variational distribution can be selected to approximate the true posterior distribution (i.e., a mixture strategy in the form of a Gaussian mixture distribution (Gaussian mixture model)). The commonly used variational distribution form is also a Gaussian mixture distribution, which can maintain the consistency of the model. The parameters of the variational distribution usually include the weight, mean, and covariance of each component.

[0070] Next, we can measure the difference between the variational distribution and the true posterior distribution by calculating the KL divergence. Specifically, the KL divergence can be decomposed into two parts: the self-information of the variational distribution and the mutual information between the variational distribution and the true posterior distribution.

[0071] In addition, a variational lower bound (Evidence Lower Bound, ELBO) can be introduced. By maximizing this lower bound, the parameters of the variational distribution can be indirectly optimized to obtain a better approximation to the true posterior distribution.

[0072] In other words, in the variational inference of the Gaussian mixture model, a variational distribution can be defined to approximate the true posterior distribution. In order to find the optimal parameters of the variational distribution, the KL divergence between the variational distribution and the true posterior distribution can be minimized. In this process, the Jhinsen inequality can be used to derive a lower bound on the KL divergence, and then the optimal parameters of the variational distribution can be found by optimizing this lower bound.

[0073] After obtaining the optimal solution for the parameters of the variational distribution, the optimal parameters of the variational distribution can be used to replace the mixed strategy in the form of a Gaussian mixture distribution to obtain the degree of difference between the strategy of the previous preset stage and the mixed strategy by calculating the KL divergence between the variational distribution and the strategy of the previous preset stage.

[0074] As an example only, the second constraint may be expressed as follows:

[0075]

[0076] in, is status information, is the state space distribution, is a very small constant.

[0077] In addition to ensuring compliance with the necessary conditions, trust region constraints are also very important. A hybrid strategy can be introduced in the trust region constraint (the second constraint), which can linearly interpolate between the current and previous iteration strategies, derive the optimal solution of the variational parameters through the Jensen inequality, and make full use of the correct direction indicated by the previous strategy. In addition, the trust region constraint ensures monotonic improvement in performance by assuming that there is a non-negative expected advantage in each state. This approach not only ensures accurate approximation of the strategy, but also does not lead to a performance degradation of the constrained Markov decision process.

[0078] This paper constructs a systematic theoretical framework to deeply explore the potential and application of Gaussian mixture models in approximating KL divergence. In addition, for the problem of mobile edge computing, a conservative policy iteration (CPI) scheme is used. During the entire iterative optimization process, the original (policy) constraints can be considered through the Gaussian mixture model, providing a solid theoretical support for evaluating and optimizing the performance of the algorithm.

[0079] In order to overcome the common problems in current mobile networks, especially the complex problems of constraint management in optimization problems and the shortcomings of traditional Lagrangian methods in dealing with constraint violations, this paper proposes a constraint strategy optimization method that integrates a pre-penalty mechanism to solve the problem of task offloading in mobile edge networks. This method can improve the robustness and efficiency of the optimization process, thereby effectively reducing constraint violations and fundamentally improving the overall optimization performance.

[0080] Specifically, the constraint strategy optimization method integrating the pre-penalty mechanism proposed in this disclosure mainly includes two contents:

[0081] First, a preemptive penalty mechanism is introduced, which can foresee and proactively respond to potential constraint violations through the barrier function. This constraint violation refers to the violation of constraints one, two, and three (i.e., the first constraint condition) in the above-mentioned constraint optimization problem. Therefore, the barrier function is used to calculate the data related to constraints one, two, and three, and is used to make punitive adjustments to the objective function value according to the first constraint condition during the training process of the secure reinforcement learning model. In addition, this first constraint condition can also consider the content related to energy consumption, delay, bandwidth, etc. during the task offloading process, and preset the constraint boundaries based on these contents.

[0082] Specifically, during the training process of the task offloading model, when the decision variables (strategies) are about to violate the constraints, the barrier function can give a higher penalty value, so that these violating solution strategies are no longer favored in the optimization process.

[0083] According to an exemplary embodiment of the present disclosure, the barrier function is used to punitively adjust the objective function value through a larger penalty term when the execution result of the current strategy of the safety reinforcement learning model is closer to the first constraint condition.

[0084] Specifically, in each iteration, the barrier function can detect whether the current solution is close to the constraint boundary. If a potential violation risk is found, the barrier function can increase the penalty intensity, prompting the optimization algorithm to adjust the direction of the solution to stay within the constraint range.

[0085] Among them, as an example only, the process of using an obstacle function to detect whether the current solution is close to the constraint boundary can be as follows: a solution (that is, a strategy) can be calculated based on the penalty function (that is, the obstacle function), and this solution is substituted into the first constraint condition (that is, the constraint condition of the optimization problem), and it is determined whether the solution tends to violate the first constraint condition after being substituted. For example, when the distance between the solution and the first constraint condition approaches a threshold or is within the threshold, it can be determined that the solution tends to violate the first constraint condition.

[0086] As an example only, the barrier function It can be expressed in the following form:

[0087]

[0088]

[0089] in, is an extended continuous convex logarithmic barrier function, is a given constant of the logarithmic barrier function, is the index of the first constraint, is the constraint function, For action, is the advantage function.

[0090] This mechanism proactively guides the optimization algorithm to avoid potential violations by presetting the boundaries of the constraints in the model, rather than the traditional Lagrangian method of punishing the solution after it violates the constraints. This forward-looking approach allows the optimization process to avoid developing in the direction of violation during the learning phase, thereby fundamentally reducing the possibility of constraint violations. At the same time, through a carefully designed penalty function, the method enables the model to evaluate the distance between the current solution and the constraint boundary in real time, and impose appropriate penalties based on the evaluation results to ensure that the solution always remains in the feasible domain or moves close to it.

[0091] The second is to embed this preemptive penalty mechanism directly into the objective function, achieving a seamless integration of the penalty term and the objective function. The objective function can guide the agent's learning strategy by maximizing the cumulative reward, which is the sum of the rewards obtained by the agent after selecting a series of actions. That is, the safe reinforcement learning model is trained by maximizing the objective function that includes the obstacle function.

[0092] Specifically, when the current solution is close to the boundary of the constraint, the barrier function can calculate a penalty term, which can reflect the degree of the current dissociation of the feasible solution. That is, the closer the current solution is to the boundary of the constraint, the larger the value of the penalty term.

[0093] By adding the penalty term to the objective function, a new optimization objective can be formed. This new objective function not only takes into account the original reward or utility, but also makes the solution that is about to violate the constraint less superior by adding the penalty term.

[0094] During the optimization process, when calculating the gradient of the objective function, the penalty term can have an impact on the value of the gradient. When the solution is close to the constraint boundary, the penalty term increases, causing the gradient of the objective function to also increase. This means that the optimization algorithm will receive a stronger signal indicating that the current solution is not suitable and needs to be adjusted.

[0095] Due to the change in gradient information, the optimization algorithm will adjust in a direction that is conducive to reducing penalties when updating the solution. This adjustment makes the algorithm tend to choose feasible solutions that are farther away from the constraint boundary, thereby effectively avoiding the risk of violating the constraint.

[0096] In this way, the model can continuously obtain effective gradient information, ensuring that it can move in the right direction at each step of optimization, and ultimately find a solution that maximizes the objective function without violating the constraints.

[0097] Just as an example, the objective function including the barrier function can be expressed as:

[0098]

[0099]

[0100] in, is an extended continuous convex logarithmic barrier function, Incorporating the expected form of the advantage function into the objective function helps reduce the variance of the estimate, provides more stable gradient information when updating the strategy, can provide more accurate action estimates than the Q function (Action-Value Function), improves the efficiency and accuracy of learning, and exhibits greater stability in uncertainty and noise.

[0101] In addition, the advantage function can capture the subtle differences between different actions in a specific state. This detailed feedback allows the agent to better adjust its strategy and thus converge to the optimal strategy faster. Moreover, the use of the advantage function enables the agent to adjust its behavior in a changing environment, enhancing the flexibility and adaptability of learning.

[0102] According to an exemplary embodiment of the present disclosure, based on the lower bound of the KL divergence obtained in the previous step, a forward-looking constraint strategy based on pre-punishment can be further introduced through an objective function containing a barrier function. The core of this strategy is to further use the barrier function method to intelligently identify and pre-punish potential constraint violations, thereby guiding the optimization process of the task offloading model to evolve in a direction that is more in line with the constraint conditions. In addition, through the conservative strategy iteration mechanism of the present disclosure, the penalty intensity can be dynamically adjusted in each iteration step to ensure that the optimization process can strictly comply with all pre-set constraints while maintaining high efficiency.

[0103] Now combine Figure 3 This paper introduces the framework of a safe reinforcement learning task offloading algorithm based on a penalty mechanism.

[0104] Figure 3 A flowchart of a method for offloading a secure reinforcement learning task based on a penalty mechanism in an exemplary embodiment of the present disclosure is shown.

[0105] Reference Figure 3 According to the exemplary embodiments of the present disclosure, some constraints (first constraints) may be set according to actual application scenarios, such as wireless communication task offloading scenarios, such as at least one of delay, bandwidth, and energy consumption. According to the pre-set constraints, a pre-recruitment mechanism may be implemented, that is, these constraints may be incorporated into the objective function to form a new optimization target. This process may be referred to as "pre-recruitment".

[0106] The original objective function can be the initial input of reinforcement learning, that is, the objective function to be optimized. The penalty function can be set by setting the first constraint condition and embedded into the objective function to realize the pre-penalty mechanism.

[0107] First, the algorithm can collect and store past states, actions, and reward information through the experience replay buffer. Then, for each iteration, sampling can be performed from the experience replay buffer to generate new observations and states by executing new action decisions to form a complete trajectory. Based on the current state and learned policy parameters, the policy network can update the network parameters with the help of the loss function through neural network methods such as policy gradient updates, and generate new action decisions. It can be understood that the policy network is the main part of the secure reinforcement learning model.

[0108] For the newly generated action decision, the trust domain constraint can be performed through KL divergence judgment. If the (trust domain) constraint is not met, the experience replay buffer can be resampled and the policy network can be updated. The training can be terminated under the condition that the (trust domain) constraint is met and the objective function of maximizing the embedded penalty function is met. In this way, the strategy can be gradually optimized while ensuring that the strategy changes smoothly.

[0109] Among them, the relevant performance indicators, such as reward value, constraint satisfaction, etc., can be calculated based on the newly generated trajectory, and the calculated indicators can be used to estimate the reward function and advantage function. The advantage function can more accurately evaluate the relative value of the current action.

[0110] In addition, according to the newly generated trajectory, the network parameters can be updated under the guidance of the loss function through two value networks (state value network, action value network).

[0111] Specifically, the goal of the state-value network is to learn a mapping function that can output the state value of the state based on the current state. The state value represents the expected return that can be obtained from the state according to the current strategy. A better strategy can be selected by comparing the state values ​​under different strategies.

[0112] The goal of the action value network is also to learn a mapping function that can output the value of the state-action pair based on the current state and action. The action value represents the expected return that can be obtained according to the current strategy after executing the action from the state. When calculating the action value, the action value network can receive the current state and action as input (sometimes it can only receive the state as input and select the action inside the network), and calculate the output value through the forward propagation of the network. The output value is the value estimate of the current state-action pair.

[0113] For state-value networks, the commonly used loss function is the mean squared error (MSE), which calculates the square difference between the state value estimate output by the network and the true state value (or target state value).

[0114] For action-value networks, a commonly used loss function is the Temporal-Difference (TD) loss function. The TD loss function calculates the squared difference between the action-value estimate output by the network and the TD target value. The TD target value is usually calculated based on the current reward and the action-value estimate of the next state (provided by the target network).

[0115] The loss function used to update the policy network can be related to the advantage function. Specifically, the loss function used in the policy network optimization can be obtained by transforming the objective function embedded in the penalty function. For example, the loss function can be in the form ( )=- [ ( , )log ( | )], where the advantage function is ( , ), which represents the average value of a specific action relative to the state. During the calculation process, the advantage function is first calculated through the state value network and the action value network, and then the current policy network is used to estimate the value of the action in the state. Next select action Finally, these values ​​are substituted into the loss function formula for updating the policy network. The Monte Carlo method or the time difference method can be used to approximate the expected value, thereby optimizing the parameters of the policy network.

[0116] The reward function is one of the foundations of advantage function calculation, which defines the benefits or costs of decisions at each time point. In the task offloading problem, the reward function can include a combination of indicators such as task execution delay, energy consumption, resource utilization, etc. The action value function (Q function) and the state value function (V function) are both defined based on the reward function. The advantage function A(s, a) is calculated based on the difference between the state value function V(s) and the action value function Q(s, a), that is, A(s,a) = Q(s, a) - V(s). Among them, V(s) represents the expected return of following the current strategy in state s, and Q(s, a) represents the expected return after executing action a in state s. In short, the advantage function evaluates the advantage of an action by comparing the difference between the Q function and the V function, and the calculation of the Q function and the V function depends on the reward function.

[0117] During the training process, based on the objective function embedded in the penalty function, the parameters (of the policy network) can be updated through the optimization method of gradient descent, and it is checked whether the objective function embedded in the penalty function converges. If so, the optimization is terminated, otherwise, the parameters are updated by gradient descent.

[0118] After multiple rounds of iterative updates, the training of the security reinforcement learning model can be completed, and the parameters of the final optimized policy network can be output.

[0119] Return to reference Figure 2 , in step 203, the state information is input into the security reinforcement learning model to obtain a decision result for the computing service request.

[0120] Among them, the security reinforcement learning model is trained by maximizing the objective function containing the obstacle function. The obstacle function is used to punitively adjust the objective function value according to the first constraint during the training process of the security reinforcement learning model. The first constraint is a constraint on at least one of energy consumption, delay, and bandwidth. The decision result is to determine which server among multiple servers the task corresponding to the computing service request will be offloaded to for processing. The multiple servers include a cloud server and at least one edge server.

[0121] In step 204, the task corresponding to the computing service request is unloaded based on the decision result.

[0122] The optimized parameters of the policy network can be applied to actual wireless communication task offloading scenarios to obtain decision results and perform task offloading based on the decision results.

[0123] According to the exemplary embodiments of the present disclosure, the present disclosure integrates the penalty term (obstacle term) into the objective function to construct an early warning system that can perceive and respond to the approach of the constraint boundary in advance, which can not only simplify the solution structure of the optimization problem, but also when the strategy gradually approaches the constraint boundary during the optimization process, the obstacle function will automatically trigger the pre-punishment mechanism to make forward-looking punitive adjustments to potential violations, thereby effectively avoiding the occurrence of violations. This forward-looking constraint strategy optimization method with a preemptive penalty mechanism not only significantly reduces the risk of actual violation of constraints, but also greatly improves the security and reliability of the strategy optimization process.

[0124] Moreover, during the optimization process, the method that incorporates the penalty term into the objective function can show a strict positive gradient. When the current solution is close to violation, the penalty term calculated by the penalty function is directly applied to the objective function, so that the model can continue to obtain effective gradient information, the gradient increases, and thus guides the search direction, ensuring that the optimization process can proceed smoothly. This avoids the zero gradient problem that may occur in the traditional Lagrangian method, that is, when the current solution approaches the constraint boundary, the optimization stagnates due to the disappearance of gradient information.

[0125] According to the disclosed secure reinforcement learning task offloading method based on pre-penalty mechanism, a pre-penalty mechanism can be introduced, which can integrate the obstacle term into the objective function, realize the forward-looking constraint strategy optimization technology, so that when the task processing approaches the resource boundary, the objective function in the task offloading model training process is punished. And in theory, this method has asymptotic convergence. In addition, in order to improve the efficiency of task offloading, a hybrid strategy can be used to guide the resource allocation and scheduling process. This framework provides a robust solution for task offloading.

[0126] Compared with solutions in related technical fields, the secure reinforcement learning method based on the pre-punishment mechanism proposed in the present disclosure, namely, the mobile edge network task offloading control scheme, has shown significant advantages and values ​​in reducing constraint violations, improving learning security, enhancing adaptability and generalization capabilities, and improving optimization efficiency and performance.

[0127] Figure 4 A block diagram of a device for offloading secure reinforcement learning tasks based on a penalty mechanism in an exemplary embodiment of the present disclosure is shown.

[0128] Reference Figure 4 , an exemplary embodiment of the present disclosure also provides a penalty mechanism-based secure reinforcement learning task unloading device 400 , which may include but is not limited to a receiving unit 401 , an acquiring unit 402 , a decision unit 403 , and an unloading unit 404 .

[0129] The receiving unit 401 may receive a computing service request from a user terminal.

[0130] The acquisition unit 402 may acquire the current status information of the user terminal.

[0131] The decision unit 403 can input the state information into the security reinforcement learning model to obtain a decision result for the computing service request, wherein the security reinforcement learning model is trained by maximizing the objective function containing the obstacle function, and the obstacle function is used to punitively adjust the objective function value according to the first constraint during the training process of the security reinforcement learning model. The first constraint is a constraint on at least one of energy consumption, delay, and bandwidth. The decision result is to determine which server among the multiple servers the task corresponding to the computing service request will be offloaded to for processing, and the multiple servers include a cloud server or an edge server among multiple edge servers.

[0132] The unloading unit 404 may unload the task corresponding to the computing service request based on the decision result.

[0133] According to an exemplary embodiment of the present disclosure, the barrier function is used to punitively adjust the objective function value through a larger penalty term when the execution result of the current strategy of the safety reinforcement learning model is closer to the first constraint condition.

[0134] According to an exemplary embodiment of the present disclosure, during the training process of the security reinforcement learning model, the degree of difference between the strategy of the previous preset stage and the hybrid strategy of the security reinforcement learning model is also limited based on a second constraint condition, the second constraint condition is a trust domain constraint, and the hybrid strategy is obtained by linearly weighting the current strategy of the security reinforcement learning model and the strategy of the previous preset stage.

[0135] According to an exemplary embodiment of the present disclosure, the degree of difference between the strategy of the previous preset stage and the hybrid strategy is calculated by KL divergence.

[0136] According to an exemplary embodiment of the present disclosure, the current strategy and the strategy of the previous preset stage are both Gaussian distributions, and the mixed strategy is a Gaussian mixed distribution.

[0137] According to an exemplary embodiment of the present disclosure, the steps for calculating the degree of difference between the strategy of the previous preset stage and the hybrid strategy may include but are not limited to: determining the corresponding variational distribution based on the Gaussian mixture distribution state of the hybrid strategy, wherein the parameters of the variational distribution are optimizable; using the Jensen inequality to perform a lower bound estimate on the KL divergence between the hybrid strategy and the variational distribution to obtain the optimal solution for the parameters of the variational distribution; calculating the KL divergence between the strategy of the previous preset stage and the optimal solution for the parameters of the variational distribution to obtain the degree of difference between the strategy of the previous preset stage and the hybrid strategy.

[0138] It can be understood that in the exemplary embodiment of the above-mentioned security reinforcement learning task unloading device 400 based on the penalty mechanism, the specific implementation process is roughly the same as the exemplary embodiment of the above-mentioned security reinforcement learning task unloading method based on the penalty mechanism, and will not be repeated here. The security reinforcement learning task unloading device 400 based on the penalty mechanism can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices can also be uniformly executed by components in physical entity devices (for example, processors, clients or servers, etc.).

[0139] Figure 5 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0140] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502, wherein the at least one memory 501 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 502, a penalty mechanism-based secure reinforcement learning task unloading method according to an exemplary embodiment of the present disclosure is executed.

[0141] As an example, the electronic device 500 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 500 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0142] In the electronic device 500, the processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0143] The processor 502 may execute instructions or codes stored in the memory 501, wherein the memory 501 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0144] The memory 501 may be integrated with the processor 502, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 501 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 502 can read files stored in the memory.

[0145] In addition, the electronic device 500 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 500 may be connected to each other via a bus and / or a network.

[0146] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned penalty-based secure reinforcement learning task offloading method.

[0147] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0148] Another embodiment of the present disclosure relates to a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned penalty-based secure reinforcement learning task offloading method.

[0149] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.

[0150] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0151] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.

[0152] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which, when executed by a processor, implements the penalty mechanism-based secure reinforcement learning task offloading method as described in any one of the above.

[0153] According to the penalty-based secure reinforcement learning task offloading method, device, electronic device, storage medium, system and computer program product provided by the present disclosure, the concept of obstacle function is introduced and integrated into the pre-punishment mechanism. The core of this mechanism is to build an early warning system that can perceive and respond to the approach of constraint boundaries in advance by integrating obstacle terms into the objective function. When the strategy gradually approaches the constraint boundary during the optimization process, the obstacle function can automatically trigger the pre-punishment mechanism and make forward-looking punitive adjustments to potential violations, thereby effectively avoiding the occurrence of violations.

[0154] In addition, through the penalty function, the system can evaluate the distance between the current solution and the constraint boundary in real time, and impose appropriate penalties based on the evaluation results to ensure that the solution always remains within the feasible domain or approaches it.

[0155] In addition, complex policy spaces can be simulated and approximated by constructing hybrid policies.

[0156] In addition, by using the secure reinforcement learning method to introduce the KL (Kullback-Leibler) divergence constraint, the optimization of the current strategy can be controlled so that the difference between the current strategy and the previous strategy is not too large.

[0157] In addition, by using variational approximation technology and deriving the optimal solution of the variational parameters through the Jensen inequality, we can more accurately calculate and give a lower bound estimate of the KL divergence under the Gaussian mixture model.

[0158] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A secure reinforcement learning task offloading method based on a penalty mechanism, characterized in that: The method comprises: Receive computing service requests from the user; Obtaining current status information of the user terminal; Inputting the state information into a secure reinforcement learning model to obtain a decision result for the computing service request, wherein the secure reinforcement learning model is obtained by training an objective function that maximizes an objective function including a barrier function, and the barrier function is used to perform a punitive adjustment on the objective function value according to a first constraint during the training of the secure reinforcement learning model, and the first constraint is a constraint for at least one of energy consumption, delay, and bandwidth, and the decision result is determining to which server among multiple servers the task corresponding to the computing service request is to be offloaded for processing, and the multiple servers include a cloud server and at least one edge server; Unloading the task corresponding to the computing service request based on the decision result; The barrier function is used to punitively adjust the objective function value through a larger penalty term when the execution result of the current strategy of the security reinforcement learning model is closer to the first constraint condition; The barrier function may be expressed as follows: in, is the barrier function, is an extended continuous convex logarithmic barrier function, is a given constant of the logarithmic barrier function, is the index of the first constraint, is the current strategy to be optimized, is the constraint function, For action, is the advantage function.

2. The method for unloading secure reinforcement learning tasks based on a penalty mechanism as claimed in claim 1, characterized in that: During the training process of the security reinforcement learning model, the degree of difference between the strategy of the previous preset stage and the hybrid strategy of the security reinforcement learning model is also limited based on a second constraint condition, wherein the second constraint condition is a trust domain constraint, and the hybrid strategy is obtained by linearly weighting the current strategy of the security reinforcement learning model and the strategy of the previous preset stage.

3. The method for unloading secure reinforcement learning tasks based on a penalty mechanism as claimed in claim 2, characterized in that: The degree of difference between the strategy of the previous preset stage and the hybrid strategy is obtained by calculating the KL divergence.

4. The method for unloading secure reinforcement learning tasks based on a penalty mechanism as claimed in claim 3, characterized in that: The current strategy and the strategy of the previous preset stage are both Gaussian distributions, and the mixed strategy is a Gaussian mixed distribution.

5. The method for unloading secure reinforcement learning tasks based on a penalty mechanism as claimed in claim 4, characterized in that: The step of calculating the degree of difference between the strategy of the previous preset stage and the mixed strategy comprises: Determining a corresponding variational distribution based on a Gaussian mixture distribution state of the mixing strategy, wherein parameters of the variational distribution are optimizable; Using the Jensen inequality to estimate the lower bound of the KL divergence between the hybrid strategy and the variational distribution, the optimal solution of the parameters of the variational distribution is obtained; The KL divergence between the strategy of the previous preset stage and the optimal solution of the parameters of the variational distribution is calculated to obtain the degree of difference between the strategy of the previous preset stage and the hybrid strategy.

6. A secure reinforcement learning task unloading device based on a penalty mechanism, characterized in that: The device comprises: The receiving unit is configured to: receive a computing service request from a user terminal; An acquiring unit is configured to: acquire current status information of the user terminal; A decision unit is configured to: input the state information into a security reinforcement learning model to obtain a decision result for the computing service request, wherein the security reinforcement learning model is obtained by training an objective function that maximizes an objective function including a barrier function, and the barrier function is used to perform a punitive adjustment on the objective function value according to a first constraint during the training of the security reinforcement learning model, and the first constraint is a constraint for at least one of energy consumption, delay, and bandwidth, and the decision result is to determine to which server among multiple servers the task corresponding to the computing service request is to be offloaded for processing, and the multiple servers include a cloud server or an edge server among multiple edge servers; an unloading unit, configured to: unload the task corresponding to the computing service request based on the decision result; The barrier function is used to punitively adjust the objective function value through a larger penalty term when the execution result of the current strategy of the security reinforcement learning model is closer to the first constraint condition; The barrier function may be expressed as follows: in, is the barrier function, is an extended continuous convex logarithmic barrier function, is a given constant of the logarithmic barrier function, is the index of the first constraint, is the current strategy to be optimized, is the constraint function, For action, is the advantage function.

7. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the secure reinforcement learning task offloading method based on the penalty mechanism as described in any one of claims 1-5.

8. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by at least one processor, the at least one processor is prompted to execute the penalty mechanism-based secure reinforcement learning task offloading method as described in any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the penalty mechanism-based secure reinforcement learning task offloading method as described in any one of claims 1 to 5 is implemented.

Images