Distributed task offloading method and system based on fault awareness and reinforcement learning

Abstract

The application relates to the technical field of edge computing, and provides a distributed task offloading method and system based on fault perception and reinforcement learning, which comprises the following steps: obtaining an offloaded task of a terminal user of an Internet of Things (IoT) that has autonomous decision-making power and offloads the task to an edge node; constructing a link number control method based on reinforcement learning; adaptively deciding an optimal offloading node number of the offloaded task according to the link number control method based on reinforcement learning; selecting a target node with optimal transmission performance according to the optimal offloading node number by cyclically calculating and comparing expected transmission times of each candidate link; establishing a target transmission link according to the target node; and dynamically adjusting the target transmission link according to the congestion degree of the transmission link. The application effectively improves the overall efficiency and robustness of task offloading of the Internet of Things, adapts to a large-scale dynamically changing Internet of Things network environment, and provides more efficient and reliable technical support for terminal task offloading under edge computing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing, and particularly relates to a distributed task offloading method and system based on fault perception and reinforcement learning. BACKGROUND

[0002] With the proliferation of Internet of Things devices, such as wearable devices, smart phones, etc., a large number of computing-intensive and delay-sensitive applications have emerged, such as face recognition, augmented reality, etc. Mobile edge computing has become one of the ideal solutions for processing these high network bandwidth demand applications due to its powerful computing capability and ultra-low delay. However, computing-intensive and delay-sensitive applications usually run in a wireless network environment. Due to the dynamic nature of the wireless environment, the transmission link is unstable, which worsens the competition among edge terminals and can cause node overload, ultimately reducing resource utilization.

[0003] In order to reduce terminal competition and improve resource utilization, using the cooperation among edge nodes to improve task offloading efficiency is a feasible idea. It mainly includes centralized method and distributed method. The centralized method needs global knowledge of system state to make task offloading decisions for each task in the edge network. However, for rapidly changing information such as environmental information, task information and workload of edge nodes, the latest global knowledge will require an overwhelming transmission overhead, which is unrealistic in a large-scale network. Traditional distributed heuristic methods are based on greedy optimization, which may fall into suboptimal in the long run. Reinforcement learning methods based on traditional methods can adaptively make task offloading decisions according to dynamic environment, and aim to maximize the discounted cumulative reward, which can avoid the occurrence of suboptimal problem, and is a potential technology that can be applied to large-scale networks compared with centralized and traditional distributed methods. However, when the existing reinforcement learning method is directly applied to select offloading strategy, it will have the problems of large strategy space and slow convergence speed of algorithm.

[0004] Nodes in mobile edge cloud often have high heterogeneous failure rates and recovery rates due to harsh environments. Frequent failures can interfere with the transmission and processing of task data, significantly increasing the risk of task failure. Existing researches optimize task offloading efficiency in ideal fault-free environments, which improves performance but is difficult to deploy in practice. Therefore, passive fault-tolerant mechanisms are introduced to improve the success rate of tasks in fault environments, but this strategy is a post-repair, and the loss caused by the fault is often irreversible. The active fault-tolerant idea tries to alleviate the impact of faults before they occur. However, these methods are mostly based on oversimplified models, ignoring key characteristics of real networks, resulting in that the task offloading problem in real fault-prone scenarios has not been effectively solved.

[0005] Therefore, when designing a distributed task offloading algorithm, the following challenges are faced. How does the user perceive and dynamically adapt to fluctuations in the wireless environment to maximize task transmission rate, how does the user make effective decisions in the absence of others' strategy information (such as privacy restrictions) when wireless links interfere with each other, how to efficiently optimize when the strategy space is large (such as the combination of node selection and power allocation) and the strategy switching is limited, and how does the user design the offloading strategy to reduce the impact of the failure of the edge node that may occur for a long time.

[0006] In view of the above considerations, current research has introduced reinforcement learning into the task offloading field, enabling the agent to autonomously learn the optimal strategy in an environment with unknown benefits. However, such methods often face privacy concerns due to strategy sharing, or the problem of a too large strategy space and slow convergence due to factors such as power allocation. Attempts are made to protect privacy through non-shared strategies, but the core challenge of low decision efficiency has not been solved. SUMMARY

[0007] The present application aims to at least solve one of the technical problems existing in the related art. To this end, the present application provides a distributed task offloading method and system based on fault awareness and reinforcement learning, which decides the optimal number of offloading edge nodes based on reinforcement learning agents, uses an epsilon-greedy strategy for action selection and updates Q values; according to the expected transmission time, select the edge node to establish the transmission link, consider the influence of the edge node failure rate; according to the network congestion degree, dynamically adjust the number of links, remove the low efficiency redundant link, the present application decouples the link number decision and the edge node selection, reduces the strategy space dimension, improves the convergence speed, at the same time, comprehensively considers the edge node failure and the network congestion, significantly improves the task offloading success rate and the system transmission efficiency.

[0008] The present application provides a distributed task offloading method based on fault awareness and reinforcement learning, comprising:

[0009] S1: obtaining an offloading task of an Internet of Things terminal user with autonomous decision-making right offloading tasks to edge nodes;

[0010] S2: constructing a link number control method based on reinforcement learning, and adaptively deciding the optimal number of offloading edge nodes of the offloading task according to the link number control method based on reinforcement learning;

[0011] S3: constructing an expected transmission time selection model based on fault awareness, and calculating the expected transmission time of each candidate link using the expected transmission time selection model based on fault awareness;

[0012] S4: comparing the expected transmission time of each candidate link to select the edge node with the optimal transmission performance, and obtaining the target offloading edge node;

[0013] S5: according to the optimal number of offloading edge nodes, the steps S3 and S4 are applied in a loop to obtain all target offloading edge nodes, and a target transmission link is established according to all target offloading edge nodes;

[0014] S6: dynamically adjusting the target transmission link according to the congestion degree of the transmission link.

[0015] Further, the offloading task is a minimum offloading unit that cannot be split, and the target of the offloading task is to maximize the actual transmission rate within the running time.

[0016] Further, the fault-aware expected transmission time selection model selects the most suitable edge node by calculating the expected transmission time of the task, the expected transmission time is the conditional expectation of the two cases of failure and successful transmission in the edge node transmission process, which is determined by the task data volume, the current channel transmission rate and the historical average transmission rate, and the calculation expression is:

[0017]

[0018]

[0019]

[0020] wherein, is the expected transmission time of the task to the edge node , is the expected failure of the edge node during transmission, is the expected success of the edge node transmission, is the task data volume, is the expected channel transmission rate of the edge node based on the expectation, is the average transmission rate, is the probability density distribution function of the edge node failure with respect to the time slot .

[0021] Further, the link number control method based on reinforcement learning includes:

[0022] S21: modeling the user equipment as a reinforcement learning agent;

[0023] S22: defining the action of the agent as selecting the maximum number of links, and defining the state of the agent as the joint strategy influence of other user equipment, which is indirectly represented by channel state information, and the reward function is the total number of bits transmitted within the two decision intervals.

[0024] Further, an ε-greedy policy is used for action selection, the agent selects the action with the highest Q value with probability 1-ε or performs random exploration with probability ε; the maximum Q value of the instant reward and the next state is obtained, and the Q table is updated using the time-difference method to gradually learn the optimal edge node quantity configuration under different network states, and the maximum link number is established according to the optimal edge node quantity configuration.

[0025] Further, the state of the agent is divided into a less congested state and a more congested state according to the channel load,

[0026] The less congested state is when the current channel quality is better than the historical average level.

[0027] The more congested state is when the current channel quality is lower than or equal to the historical average level.

[0028] Further, the channel quality is reflected by aggregated information, and the smoothed channel information is obtained by exponential weighted average.

[0029] Further, the aggregated information is defined as:

[0030]

[0031] wherein, is the aggregated information of the user of the edge node at the time slot , is the channel gain of the user of the edge node at the time slot , is the background noise power, is the channel gain of the user of the edge node at the time slot , is the transmission power of the user of the edge node at the time slot , is the set of all other users except the user .

[0032] Further, the calculation expression of the channel information is:

[0033]

[0034] wherein, is the channel information of the user of the edge node at the time slot , is the channel weight, for a time slot user edge node channel information, for a time slot user edge node aggregated information between.

[0035] The application also provides a distributed task offloading system based on fault awareness and reinforcement learning, which is used to execute the above-mentioned distributed task offloading method based on fault awareness and reinforcement learning, and comprises:

[0036] A task acquisition module is configured to acquire an offloading task of offloading a task to an edge node by an Internet of Things terminal user with autonomous decision-making power.

[0037] An optimal node quantity decision module is configured to construct a link quantity control method based on reinforcement learning, and to adaptively decide an optimal offloading edge node quantity of the offloading task according to the link quantity control method based on reinforcement learning.

[0038] An expected transmission time calculation module is configured to construct an expected transmission time selection model based on fault awareness, and to calculate an expected transmission time of each candidate link by using the expected transmission time selection model based on fault awareness.

[0039] A target node selection module is configured to compare the expected transmission time of each candidate link, to select an edge node with optimal transmission performance, and to obtain a target offloading edge node.

[0040] A target transmission link construction module is configured to cyclically apply steps S3 and S4 according to the optimal offloading edge node quantity, to obtain all target offloading edge nodes, and to establish a target transmission link according to the all target offloading edge nodes.

[0041] A dynamic adjustment module is configured to dynamically adjust the target transmission link according to a congestion degree of the transmission link.

[0042] The above-mentioned one or more technical solutions in the embodiments of the application have at least one of the following technical effects:

[0043] The optimal number of offloaded edge nodes is decided based on the reinforcement learning intelligent agent, an epsilon-greedy strategy is adopted for action selection and Q value updating, an edge node is selected according to the expected transmission time to establish a transmission link, the influence of the edge node failure rate is considered, the number of links is dynamically adjusted according to the network congestion degree, and the inefficient redundant links are removed. The number of links is decoupled from the selection of edge nodes, the dimension of the strategy space is reduced, the convergence speed is improved, the edge node failure and network congestion are comprehensively considered, and the task offloading success rate and system transmission efficiency are significantly improved.

[0044] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0046] Figure 1 It is a flowchart of a distributed task offloading method based on fault perception and reinforcement learning provided by the application.

[0047] Figure 2 It is the actual transmission rate under the Rayleigh distribution and the Rician distribution provided by the embodiment of the application.

[0048] Figure 3 It is the number of task completions under the Rayleigh distribution and the Rician distribution provided by the embodiment of the application.

[0049] Figure 4 It is a structure diagram of a distributed task offloading system based on fault perception and reinforcement learning provided by the application.

[0050] Reference signs:

[0051] 101, task acquisition module; 102, optimal node number decision module; 103, expected transmission time calculation module; 104, target node selection module; 105, target transmission link construction module; 106, dynamic adjustment module. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0053] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are contained in at least one embodiment or example of the embodiments of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0054] The following will be described in combination with Figures 1 to 4 A distributed task offloading method and system based on fault perception and reinforcement learning are described.

[0055] As Figure 1 shown, a distributed task offloading method based on fault perception and reinforcement learning comprises:

[0056] S1: obtaining an offloading task of a terminal user of Internet of Things with autonomous decision-making right offloading a task to an edge node;

[0057] For the terminal user of Internet of Things with autonomous decision-making right, the computing task is offloaded to the edge node. In view of the low efficiency of centralized decision-making when the terminal scale expands, and the user is unwilling to transfer the scheduling authority for privacy and security, and the improvement of the terminal itself, the implementation of distributed autonomous decision-making is jointly promoted.

[0058] The task offloading efficiency is measured by the number of successfully completed offloading tasks and the actual transmission rate of task data within a specified time;

[0059] A task offloading system composed of a plurality of terminal users UE (User Equipment) and an edge node EN (Edge Node), , wherein, for the second end user, for the first end user, for the second end user, for the first edge node, for the second edge node, for the first edge node, for the second edge node, the task offloading system operates in discrete time slots, each time slot has a length of , the power allocation policy for each user is , , , for the user in time slot the transmission power of the first edge node, for the user in time slot the transmission power of the second edge node, for the user in time slot the transmission power of the first edge node, for the user in time slot the transmission power of the second edge node; The total power constraint is satisfied:

[0060]

[0061] where,

[0062] for the user in time slot the transmission power of the edge node, the maximum transmission power of the user. The transmission rate model considers the interference between users of the same edge node, the transmission rate of user edge node is:

[0063] where,

[0064] is the transmission rate of user in time slot edge node

[0065] , is the corresponding channel bandwidth of edge node , is the background noise power, is the channel gain of user edge node , is the interference from other users of the same edge node. ​​​​​​​​​​channel gain, for a time slot user edge node transmission power, for a set of all other users except user .

[0066] In the calculation model, the offloaded task is the minimum offloading unit and cannot be split, the edge node has unlimited parallel processing capability, the target of offloading task is to maximize the actual transmission rate within the running time, and the calculation expression is:

[0067]

[0068] wherein, is the actual transmission rate, is the data size of the task of the th successful transmission of the th user equipment, is the running time, is the time slot length, is the task of successful transmission, is the total number of users.

[0069] The target is equivalent to maximizing the total amount of successfully completed tasks. The system considers the problem of resource waste caused by edge node failure and transmission interruption, and aims to improve task success rate and system efficiency by optimizing offloading strategy.

[0070] The premise of the task offloading system and method designed by the application is that the number of links is the key to improving system efficiency. In a typical edge computing scenario, when user equipment competes for the same channel, the transmission rate of a single device can be expressed as:

[0071]

[0072] wherein, is the transmission rate of a single device when user equipment competes for the same channel, is the channel width, is the transmission power of a single device, is the channel gain, background noise power.

[0073] The ratio of the transmission rate of the exclusive channel to the shared channel is:

[0074]

[0075] wherein, is the transmission rate of the exclusive channel;

[0076] Therefore, even if only two user devices share the channel, the total transmission rate will be at least one order of magnitude lower than in the case of an exclusive channel, and in view of the fact that there can be more user devices competing for the channel in a real environment, the channel of the edge node should be kept as exclusive as possible to avoid competition between user devices.

[0077] S2: Construct a link number control method based on reinforcement learning to adaptively determine the optimal number of offloading edge nodes for the offloading task according to the link number control method based on reinforcement learning;

[0078] When planning the calculation and communication resources, the traditional method converges slowly due to the large strategy space, and theoretical analysis shows that the number of links is the key to improving efficiency, so the application takes it as the core decision target of reinforcement learning.

[0079] According to the link number control method based on reinforcement learning, the optimal number of offloading edge nodes for the offloading task is adaptively determined.

[0080] Specifically, an epsilon-greedy strategy is used for action selection: the agent selects the action (i.e., the number of links) with the highest current Q value with a probability of 1-epsilon, or randomly explores with a probability of epsilon. After each decision, the algorithm updates the Q table using the temporal difference method according to the immediate reward and the maximum Q value of the next state, thereby gradually learning the optimal edge node number configuration under different network states. Through continuous interaction with the environment, the agent can adaptively adjust the number of offloading edge nodes to optimize the overall offloading efficiency.

[0081] The link number control method based on reinforcement learning adaptively determines the optimal number of offloading edge nodes for the offloading task, and is suitable for user devices in a distributed wireless communication system;

[0082] S21: Model the user device as a reinforcement learning agent;

[0083] S22: Define the action of the agent as selecting the maximum number of links, and define the state of the agent as the influence of the joint strategy of other user devices, which is indirectly represented by measurable channel state information, and the reward function is the total number of bits transmitted within the interval between two decisions.

[0084] The user device is modeled as a reinforcement learning agent, and the adaptive adjustment of the number of links is realized through interaction with the environment.

[0085] An epsilon-greedy strategy is used for action selection, and the agent selects the action with the highest current Q value with a probability of 1-epsilon, or randomly explores with a probability of epsilon; obtain the immediate reward and the maximum Q value of the next state, and update the Q table using the temporal difference method, gradually learning the optimal edge node number configuration under different network states, and establishing the maximum number of links according to the optimal edge node number configuration.

[0086] The state of the agent is divided into less congested state and more congested state according to the channel load condition,

[0087] The less congested state is when the current channel quality is better than the historical average level;

[0088] The more congested state is when the current channel quality is lower than or equal to the historical average level.

[0089] The channel quality is reflected by aggregated information, and the smoothed channel information is obtained by exponential weighted average.

[0090] To solve the problem that the strategy of other user equipment is not observable in distributed decision, the system state is indirectly represented by measurable channel state information.

[0091] Specifically, the aggregated information is defined to reflect the channel quality, and the smoothed channel information is obtained by exponential weighted average as a reliable approximation of the state input. In the decision time slot , the aggregated information can be defined as:

[0092]

[0093] wherein, is the aggregated information of the user and the edge node in the time slot , is the channel gain of the user and the edge node in the time slot , is the background noise power, is the channel gain of the user and the edge node in the time slot , is the transmission power of the user and the edge node in the time slot , and is the set of all other users except the user . is the interference plus noise power suffered by the user on the same edge node

[0094] channel, then:

[0095]

[0096] The transmission rate of the user in the time slot

[0097] ​​​

[0098] in, For time slots user edge nodes transmission rate For edge nodes The corresponding channel bandwidth;

[0099] Will Substituting into the above equation, we get:

[0100]

[0101] but:

[0102]

[0103] We can obtain:

[0104]

[0105] in, For time slots user edge nodes The transmission rate;

[0106] All parameters on the right side of the equation are known quantities, and the aggregated information can be directly calculated. The user transmission rate can be expressed as:

[0107]

[0108] in, In state space Time slot user edge nodes transmission rate For edge nodes The corresponding channel bandwidth.

[0109] This indicates that the aggregated information fully includes the time slots. The information required for transmission rate decisions can accurately characterize the impact of joint strategies. Considering the random fluctuation characteristics of the channel, an exponentially weighted average is used to eliminate random bias, defining... With edge nodes Average aggregate information:

[0110]

[0111] in, For time slots user edge nodes Channel information, For channel weights, For time slots user edge nodes Channel information, For time slots user edge nodes Aggregated information between them.

[0112] The parameter is a real number in the interval (0,1]. As an effective approximation of the state in reinforcement learning.

[0113] A significant advantage of this invention lies in the compact design of the action space. Unlike traditional methods that treat the entire discrete policy space as the action space, this invention limits the action space to the maximum number of selectable links (i.e., the number of optimal offloading edge nodes), making the size of the action space independent of the number of network nodes. This effectively avoids the curse of dimensionality in the action space and ensures the convergence speed of the algorithm.

[0114] In this invention, the agent directly determines the upper limit of the number of edge nodes that each user can simultaneously unload tasks by deciding the maximum number of links. That is, the maximum number of links is the optimal number of unloaded edge nodes in the final implementation stage.

[0115] In terms of reward function design, the reward is defined as the cumulative number of transmitted bits, which directly reflects the transmission efficiency of the system and provides a clear learning objective for the agent.

[0116] At the algorithm implementation level, a tabular Q-learning algorithm is adopted, and continuous states are approximated by tile coding. In order to achieve intelligent perception of channel congestion, the state space is divided into two categories according to the channel load: when the current channel quality is better than the historical average, it is defined as a less congested state, and vice versa.

[0117] Calculate the current state, select an action based on the ε-greedy policy, and update the Q-value function. When the decision requires reducing the number of links, the link with the worst channel quality is removed to ensure that the actual number of links matches the decision.

[0118] This invention achieves scalability and practicality in link number control in large-scale network environments through reasonable state representation, compact action space design, and clear reward mechanism, providing an effective technical solution for distributed wireless resource management.

[0119] S3: Construct a fault-aware expected transmission time selection model, and use the fault-aware expected transmission time selection model to calculate the expected transmission time of each candidate link.

[0120] After determining the number of offloaded edge nodes, a heuristic strategy is adopted to determine the target offloaded edge nodes, taking into account the failure rate of edge nodes, and a transmission link is established. The selected edge nodes must be in a healthy state to provide services. Internally, the receiving, processing and result feedback processes of tasks are managed by maintaining receiving, execution and feedback queues.

[0121] After determining the target number of links, the designed heuristic strategy iteratively calculates and compares the expected transmission time of each candidate link, continuously selecting the edge node with the best transmission performance to establish a connection until the actual number of links reaches the target value. This process ensures that, considering factors such as edge node failure rate, the system can adaptively construct an efficient set of transmission links.

[0122] The fault-aware expected transmission time selection model selects the most suitable edge node by calculating the expected transmission time of the task. The expected transmission time is the conditional expectation under two scenarios: failure and successful transmission during the edge node's transmission process. It is determined by the task data volume, the current channel transmission rate, and the historical average transmission rate. The calculation expression is as follows:

[0123]

[0124]

[0125]

[0126] in, For tasks to edge nodes Expected transmission time For edge nodes The expectation of a failure during transmission. For edge nodes Expectations for successful transmission For the amount of task data, For edge nodes Based on the expected estimated channel transmission rate, For average transmission rate, For edge node failures related to time slots The probability density distribution function.

[0127] S4: Compare the expected transmission times of each candidate link and select the edge node with the best transmission performance to obtain the target offloading edge node;

[0128] After determining the maximum number of links, the user equipment needs to allocate suitable target edge nodes for each task to be offloaded. To maximize task offloading efficiency, this invention proposes selecting edge nodes with the goal of minimizing the transmission time of a single task. The calculation of the expected transmission time comprehensively considers three key factors: the size of the task data, the fault distribution characteristics of the edge nodes, and the channel transmission rate.

[0129] According to the full expectation formula in probability theory, the expected transmission time is decomposed into conditional expectations under two scenarios: failure during edge node transmission and successful transmission. (Task to edge node) Expected transmission time It is determined by three factors: the amount of task data. Fault distribution at edge nodes (probability density function is) and the channel transmission rate based on expected estimation. The calculation expression is:

[0130]

[0131]

[0132]

[0133] in, For tasks to edge nodes Expected transmission time For edge nodes The expectation of a failure during transmission. For edge nodes Expectations for successful transmission For the amount of task data, For the channel transmission rate estimated based on expectations, For average transmission rate, For edge node failures related to time slots The probability density distribution function.

[0134] This invention further subdivides the expected transmission time into three components, based on a probabilistic analysis of the timing of failures during transmission: additional costs incurred due to potential failures at edge nodes. It is essentially the time of failure occurrence. The expected value within the failure probability range represents the time invested but lost due to failure; the cost required for data retransmission when an edge node fails. ,in For fixed retransmission time, this integral term reflects the result of weighting this cost by the probability of failure; and the cost required for successful transmission. Basic transmission time Multiply by the probability of successful transmission.

[0135] In a preferred embodiment of the invention, it is assumed that edge node failures are related to time slots. The probability density distribution follows a Poisson process, and the fault time interval follows an exponential distribution, i.e.:

[0136]

[0137] in, is the scale parameter of the exponential distribution;

[0138] Based on this assumption, a closed-form expression for the expected transmission time is derived through integration:

[0139]

[0140] The final calculation results show that the expected transmission time is indeed determined by only three variables: the amount of task data, the current channel transmission rate, and the historical average transmission rate. These parameters can all be obtained through actual measurement or estimation from historical data.

[0141] The specific execution flow of this method includes: calculating the expected transmission time for each available channel, selecting the edge node with the minimum expected transmission time to establish a link, and updating the actual link set. Through this edge node selection mechanism based on expected transmission time, this method can effectively improve the success rate of task offloading, reduce transmission latency, and thus improve the overall system performance.

[0142] S5: Apply steps S3 and S4 iteratively based on the optimal number of offloading edge nodes to obtain all target offloading edge nodes, and establish the target transmission link based on all target offloading edge nodes.

[0143] By calculating the expected transmission time of the task, the most suitable edge node is selected to establish a transmission link.

[0144] This invention provides an efficient and reliable edge node selection scheme for task offloading in distributed computing systems by establishing an accurate mathematical model and considering uncertainties in the actual network environment.

[0145] For end users, the number of links can be dynamically adjusted based on congestion levels during task execution. This mechanism is not only an immediate response to network changes but also a proactive fault-tolerance strategy designed to avoid transmission failures caused by congestion, thereby ensuring end-to-end task reliability.

[0146] S6: Dynamically adjust the target transmission link based on the congestion level of the transmission link.

[0147] When the actual number of links exceeds the optimal number for reinforcement learning decisions, the redundant links with the lowest performance are removed based on the performance metrics of each link.

[0148] To verify the effectiveness of this invention, a simulation of a mobile edge computing environment was conducted to evaluate the performance of link number control and edge node selection. The simulation system was implemented in C++ and ran on a computing platform configured with an AMD Ryzen 7 5800H processor.

[0149] In the simulation environment settings, edge nodes and user equipment are on sides of length [missing information]. The channels are uniformly distributed within a square area. The fast fading characteristics of the channel are modeled using a Ricean distribution, and the channel gain is mainly determined by the distance between devices and the path loss exponent.

[0150] Regarding edge node reliability modeling, the average failure time interval for each edge node is randomly generated to ensure that the failure rate remains within the range of [3.5%, 5%] over a one-minute runtime. The recovery time for a failed edge node is set to 100 milliseconds. The task size is simulated using a Pareto distribution with the following shape parameter: .

[0151] To reduce the impact of random factors and obtain accurate statistical results, ensuring the reliability and statistical significance of the evaluation results, all simulation data were within a 6×10⁻⁶ range. 4 The experiment was run on each time slot, and the average of 50 independent experiments was taken as the final result.

[0152] To provide sufficient evidence for verifying the effectiveness of the present invention in practical application scenarios, the present invention (Scalable Reinforcement Learning, SRL) was compared with Random Offloading (RO), Greedy Offloading (GO), and Multi-armed Bandits (MAB).

[0153] Random Offloading (RO): Performance lower limit benchmark, where user equipment randomly selects target edge nodes and allocates power;

[0154] Greedy Offloading (GO): In each time slot, the user device selects the best edge node based on a preference metric, pursuing a local optimum.

[0155] Multi-armed Bandits (MAB): User equipment selects the number of links using a gambling machine algorithm, ignoring state changes, to compare the effectiveness of state-aware designs.

[0156] This invention considers different target selection strategies and edge node states, as follows:

[0157] Average channel information (ACI) strategy: When selecting target edge nodes for user tasks, this strategy mainly considers the long-term channel quality of the link and calculates the channel state information by exponential weighted averaging. It aims to select nodes with more stable channel conditions and lighter loads and is an evaluation method based on the physical state of the link.

[0158] Expected Transmission Time (ETT) strategy: This strategy performs an end-to-end comprehensive performance evaluation when selecting target edge nodes for user tasks. It not only considers the channel transmission rate, but also introduces the edge node failure probability and task retransmission cost. It calculates the expected time required to complete the transmission through a mathematical model, aiming to select the edge node with the highest overall transmission efficiency. It is an application-oriented evaluation method.

[0159] Idealized situation (IS): Under this setting, the simulation environment ignores the failure mechanism of edge nodes and assumes that all edge nodes remain in a normal state throughout the entire operation. This assumption is used to evaluate the upper limit of the algorithm's performance in a perfect hardware environment. At the same time, in this case, since there is no need to consider edge node failures, the ACI strategy based on channel quality and the ETT strategy that comprehensively considers failures will tend to be consistent in decision-making performance.

[0160] The adaptability of this invention under different channel conditions was verified through simulation. The tests employed two typical wireless channel models: Ricean distribution and Rayleigh distribution, using the user's actual transmission rate and the number of tasks completed per unit time as performance evaluation indicators.

[0161] Simulation results show that the present invention (SRL) exhibits superior performance under both channel models. Regarding transmission rate, as... Figure 2 As shown, the present invention significantly outperforms the comparative algorithm when considering different target selection strategies (ACI and ETT) under ideal state IS and fault state. Figure 2 As shown in Figure (a), under the Rice distribution, in the three cases of ideal state (IS) and fault state using ACI or ETT strategies, the present invention shows significant improvement compared to the RO algorithm, GO algorithm, and MAB algorithm. Figure 2 As shown in Figure (b), SRL maintains a stable performance advantage even under Rayleigh distribution conditions with worse channel conditions. Regarding the number of tasks completed, as... Figure 3 As shown, SRL also demonstrates significant advantages when considering different target selection strategies, ACI and ETT, under ideal state IS and fault state conditions. Figure 3As shown in Figure (a), under the Rice distribution, in the ideal state (IS) and the fault state using the ACI or ETT strategy, this invention also shows a significant improvement over the RO algorithm, GO algorithm, and MAB algorithm. Figure 3 As shown in Figure (b), the effectiveness of SRL is further verified under Rayleigh distribution with worse channel conditions, proving that the present invention has a wide channel adaptability, can maintain stable performance in different wireless channel environments, and is suitable for various channel conditions in practical application scenarios.

[0162] like Figure 4 As shown, a distributed task offloading system based on fault perception and reinforcement learning is used to execute the aforementioned distributed task offloading method based on fault perception and reinforcement learning, including:

[0163] The task acquisition module 101 acquires the unloading tasks of IoT terminal users with autonomous decision-making power to unload tasks to edge nodes;

[0164] The optimal node number decision module 102 constructs a link number control method based on reinforcement learning, and adaptively determines the optimal number of unloading edge nodes for the unloading task based on the link number control method based on reinforcement learning.

[0165] The expected transmission time calculation module 103 constructs an expected transmission time selection model based on fault perception, and uses the expected transmission time selection model based on fault perception to calculate the expected transmission time of each candidate link.

[0166] The target node selection module 104 compares the expected transmission time of each candidate link and selects the edge node with the best transmission performance to obtain the target offloading edge node.

[0167] The target transmission link construction module 105 applies steps S3 and S4 repeatedly according to the optimal number of offloading edge nodes to obtain all target offloading edge nodes, and establishes the target transmission link based on all target offloading edge nodes.

[0168] The dynamic adjustment module 106 dynamically adjusts the target transmission link according to the congestion level of the transmission link.

[0169] Through the collaborative work of the above modules, the optimal number of offloading edge nodes is determined by the reinforcement learning agent, and an ε-greedy strategy is used for action selection and Q-value updates. Edge nodes are selected to establish transmission links based on the expected transmission time, taking into account the impact of edge node failure rates. The number of links is dynamically adjusted according to network congestion levels, and inefficient redundant links are removed. This invention decouples link quantity decision-making from edge node selection, reduces the dimensionality of the policy space, improves convergence speed, and comprehensively considers edge node failures and network congestion, significantly improving task offloading success rate and system transmission efficiency.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed task offloading method based on fault perception and reinforcement learning, characterized in that, include: S1: Obtain the unloading task from IoT terminal users with autonomous decision-making power to unload tasks to edge nodes; S2: Construct a link number control method based on reinforcement learning, and adaptively determine the optimal number of unloading edge nodes for the unloading task based on the link number control method based on reinforcement learning. Link number control methods based on reinforcement learning include: S21: Model the user equipment as a reinforcement learning agent; S22: The agent's action is defined as selecting the maximum number of links, the agent's state is defined as the influence of the joint policies of other user equipment, and is indirectly represented by channel state information. The reward function is the total number of bits transmitted between two decision intervals. S3: Construct a fault-aware expected transmission time selection model, and use the fault-aware expected transmission time selection model to calculate the expected transmission time of each candidate link. The fault-aware expected transmission time selection model selects the most suitable edge node by calculating the expected transmission time of the task. The expected transmission time is the conditional expectation under two scenarios: failure and successful transmission during the edge node's transmission process. It is determined by the task data volume, the current channel transmission rate, and the historical average transmission rate. The calculation expression is as follows: in, For tasks to edge nodes Expected transmission time For edge nodes The expectation of a failure during transmission. For edge nodes Expectations for successful transmission For the amount of task data, For edge nodes Based on the expected estimated channel transmission rate, For average transmission rate, For edge node failures related to time slots The probability density distribution function; S4: Compare the expected transmission times of each candidate link and select the edge node with the best transmission performance to obtain the target offloading edge node; S5: Apply steps S3 and S4 iteratively based on the optimal number of offloading edge nodes to obtain all target offloading edge nodes, and establish the target transmission link based on all target offloading edge nodes. S6: Dynamically adjust the target transmission link based on the congestion level of the transmission link.

2. The distributed task offloading method based on fault perception and reinforcement learning according to claim 1, characterized in that, The unloading task is the smallest indivisible unloading unit, and the goal of the unloading task is to maximize the actual transmission rate within the runtime.

3. The distributed task offloading method based on fault perception and reinforcement learning according to claim 1, characterized in that, An ε-greedy strategy is adopted for action selection. The agent selects the action with the highest current Q value with a probability of 1-ε, or explores randomly with a probability of ε. It obtains immediate reward and the maximum Q value of the next state, updates the Q table using the temporal difference method, and gradually learns the optimal configuration of the number of edge nodes under different network states. The maximum number of links is established based on the optimal configuration of the number of edge nodes.

4. The distributed task offloading method based on fault perception and reinforcement learning according to claim 1, characterized in that, The agent's state is divided into a less congested state and a more congested state based on the channel load. A less congested state is when the current channel quality is better than the historical average. A relatively congested state is defined as the current channel quality being lower than or equal to the historical average.

5. A distributed task offloading method based on fault perception and reinforcement learning according to claim 4, characterized in that, Channel quality is reflected by aggregated information, and smoothed channel information is obtained by exponential weighted averaging.

6. The distributed task offloading method based on fault perception and reinforcement learning according to claim 5, characterized in that, Aggregated information is defined as: in, For time slots user edge nodes Aggregated information, For time slots user edge nodes Channel gain, Background noise power, For time slots user edge nodes Channel gain, For time slots user edge nodes Transmission power, In addition to users The collection of all other users besides [the user group].

7. A distributed task offloading method based on fault perception and reinforcement learning according to claim 5, characterized in that, The expression for calculating channel information is: in, For time slots user edge nodes Channel information, For channel weights, For time slots user edge nodes Channel information, For time slots user edge nodes Aggregated information between them.

8. A distributed task offloading system based on fault perception and reinforcement learning, characterized in that, To execute a distributed task offloading method based on fault perception and reinforcement learning as described in any one of claims 1 to 7, comprising: The task acquisition module acquires the unloading tasks of IoT terminal users with autonomous decision-making power to unload tasks to edge nodes; The optimal node number decision module constructs a link number control method based on reinforcement learning, and adaptively determines the optimal number of unloading edge nodes for the unloading task based on the link number control method based on reinforcement learning. The expected transmission time calculation module constructs an expected transmission time selection model based on fault perception and uses the expected transmission time selection model based on fault perception to calculate the expected transmission time of each candidate link. The target node selection module compares the expected transmission time of each candidate link and selects the edge node with the best transmission performance to obtain the target offloading edge node. The target transmission link construction module applies steps S3 and S4 iteratively according to the optimal number of offload edge nodes to obtain all target offload edge nodes, and establishes the target transmission link based on all target offload edge nodes. A dynamic adjustment module that dynamically adjusts the target transmission link based on the congestion level of the transmission link.

Images