A network tomography-based edge task offloading scheduling method and device

Abstract

A network tomography-based edge task offloading scheduling method and device, the method comprising: after the edge device requests to offload the computing task to the edge server, predicting the communication time delay of the task offloading by using network tomography, and constructing a mathematical model for depicting the end-to-end time delay of the edge task offloading. Then, the trained reinforcement learning model method is used for task offloading and scheduling. The present application generates a series of curriculum environment to assist the training of reinforcement learning intelligent agent by using curriculum learning method, accurately predicts the network communication time delay, constructs the mathematical model of the end-to-end time delay of the edge computing task, and uses reinforcement learning for task scheduling and curriculum generation of reinforcement learning training environment based on curriculum learning, which more accurately depicts the end-to-end time delay of the edge computing task offloading, so that the offloading request of the edge task can be better scheduled in the complex and changeable edge network environment, and a lower total task end-to-end time delay is obtained.

Description

Technical Field

[0001] The present invention relates to a task scheduling method and device based on network fault perspective. This invention can schedule edge computing tasks in complex, dynamically fluctuating edge scenarios based on the principle of minimizing the end-to-end computing latency of edge tasks. Background Art

[0002] The emergence of cloud computing provides an efficient computing platform for various data and applications. However, the current development of network bandwidth and network transmission technology has far outstripped the growth of data. Cloud computing results in excessively long transmission links, which in turn leads to higher latency and security risks. To address these shortcomings of cloud computing, edge computing has emerged. In edge computing, computing resources are placed closer to data sources and end devices, providing high-bandwidth, low-latency services. Network edge servers also have sufficient computing power to process source data locally. By sending data to edge servers rather than cloud servers for computation, bandwidth pressure during network transmission can be reduced, accelerating data analysis and processing.

[0003] In edge computing, resource-rich edge servers are deployed near data sources and end devices to provide high-bandwidth, low-latency services. As an extension of the cloud, the edge can provide a variety of capabilities, including device access, data processing, and AI intelligence. Many industrial efforts have also attempted to offload tasks to edge servers, such as Alibaba Cloud ENS and Google IoT. With the development of edge computing, the scheduling problem of edge task offloading has been widely studied. Therefore, efficiently dispatching tasks to appropriate edge servers in dynamic edge networks is crucial for improving task execution efficiency.

[0004] Edge computing task offloading scheduling methods determine the offloading schedule based on the end-to-end latency of edge tasks. End-to-end latency typically refers to the sum of the communication latency when offloading edge computing tasks to edge servers and the computational latency when executing the tasks on the edge servers. This means that when scheduling edge tasks, a scheduling scheme should be designed to minimize the total end-to-end latency of edge tasks. Existing scheduling methods tend to assume that the network between users and edge devices is static and stable.

[0005] Unfortunately, existing edge task offloading scheduling methods struggle to accurately capture the communication latency between clients and servers when the network dynamically fluctuates. This is because they often overlook the impact of network fluctuations on task scheduling strategies. Specifically, the complexity and variability of real-world edge networks lead to frequent changes in edge network topology. Furthermore, the communication latency between edge servers in the real world also varies significantly, making existing edge computing task scheduling methods unsuitable for task scheduling in dynamic edge network scenarios. Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a method and device for scheduling edge tasks based on network fault perspective.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a method for scheduling edge task offloading based on network fault perspective, comprising the following steps:

[0008] (1) Network delay analysis based on network tomography. The scheduling of edge computing tasks depends on obtaining the communication delay of edge task offloading. This method proposes a responsive network tomography scheme. When scheduling edge computing tasks, it heuristically detects the communication delay between edge users and adjacent clients to estimate the communication delay of edge task offloading.

[0009] (2) Construct a mathematical model for edge task delay. When scheduling edge computing tasks, it is necessary to minimize the sum of the end-to-end delays of all edge tasks (that is, the sum of the network communication delay caused by task offloading and the computational delay of edge tasks on the edge server). To this end, a mathematical model that describes the end-to-end delay of edge tasks is established before scheduling edge computing tasks.

[0010] (3) Reinforcement Learning-Based Task Offloading Scheduling. Edge task offloading scheduling uses a reinforcement learning-based strategy. This paper employs a “critic-only” reinforcement learning algorithm. The agent’s goal is to maximize the cumulative discounted reward, where the reward is the end-to-end delay of the task, the state of the environment is the current task schedule, and the agent’s action is to change the task scheduling strategy to maximize the reward.

[0011] (4) Curriculum-based reinforcement learning optimization strategy. Considering that reinforcement learning often performs poorly and has difficulty generalizing in network environments with complex and fluctuating environmental distributions, this invention uses curriculum learning to create courses to improve reinforcement learning training. The reinforcement learning agent continuously learns in the course environment created by the curriculum learning method, achieving better training results than directly training the reinforcement learning agent.

[0012] Wherein, step (1) specifically includes:

[0013] 1.1) Step S11: When an edge user requests to offload a task for the first time, the method probes the network to analyze the communication delay from the client to each server.

[0014] 1.2) Step S12: When the edge user next requests to offload a task, this method first proactively collects latency information from the K (set to 5 based on experience in this method) nearest servers and determines whether network transmission has caused a significant change in communication latency compared to the last time the task was requested. This is because, even if there are network fluctuations, these fluctuations are limited in a short period of time, and the latency to the nearest server is the best indicator of network fluctuations.

[0015] 1.3) Step S13: If the relative order of the delay values ​​between servers changes significantly compared to the last time, or any delay value deviates from the threshold (set to 10% based on experience in this method), it is considered necessary to re-probe the server network to analyze the communication delay when offloading edge tasks to different edge servers.

[0016] Wherein, step (2) specifically includes:

[0017] 2.1) The variables involved in the mathematical model are as follows:

[0018] M, namely the number of edge servers, has a data type of a positive integer and describes the number of servers that can be used to provide edge computing services to different edge users in this method.

[0019] N, namely the number of edge users, has a data type of a positive integer and describes the number of edge users that can initiate edge task offloading requests to the edge server in this method.

[0020] T n,k , which is the end-to-end task delay when edge user n requests to offload edge computing task k to the edge server, or the total delay incurred when the edge task is offloaded to the edge server for computation. It numerically equals the communication delay during the edge task offloading process and the computation delay on the edge server after the edge computing task is offloaded to the edge server.

[0021] H n,k , that is, the end-to-end task delay threshold when edge user n requests to offload edge computing task k to the edge server, that is, the longest end-to-end delay that the edge task can tolerate.

[0022] The data type is Boolean and is used to describe whether the edge computing task k from edge user n is offloaded to edge server m.

[0023] The data type is Boolean, which is used to describe whether the edge server m can provide computing services for the edge computing task k.

[0024] Used to describe the communication delay between edge server m and edge user n caused by offloading edge computing tasks, obtained from network fault perspective.

[0025] Used to describe the computing delay between edge server m and edge user n caused by offloading edge computing tasks, which is predicted by the server.

[0026] c n,k , is used to describe the amount of computing resources requested by edge user n to the server providing edge computing services when initiating an offloading request for edge computing task k.

[0027] TH resource ,The data type is percentage data, which is used to describe the threshold of computing resource occupancy rate on the edge server. In order to meet its own needs and some sudden computing task requirements, the edge computing server may need to reserve some computing resources. Therefore, when the current computing resource occupancy rate on the edge server is higher than the threshold TH resource When the edge computing task newly assigned to the edge server will wait for computing services to be provided until the computing resource occupancy rate on the server drops below the threshold.

[0028] C m , used to describe the total amount of computing resources on edge server m.

[0029] 2.2) Mathematical models used to describe the end-to-end latency after edge computing task offload include:

[0030] 2.2.1) The optimization goal of the task offloading scheduling strategy is to minimize the sum of the end-to-end delays of all edge computing tasks from edge users, that is:

[0031]

[0032] The offloading scheduling problem of edge computing tasks is modeled as a minimization problem, which is subject to the following constraints;

[0033] 2.2.2) Constraint 1: The end-to-end delay of an edge computing task k from edge user n being offloaded to edge server m is equal to the sum of the communication delay of the task being offloaded to the edge server and the computation delay of the task on the edge server:

[0034]

[0035] 2.2.3) Constraint 2: The end-to-end delay of edge computing task k from edge user n when it is offloaded to edge server m should be lower than the delay threshold of the edge computing task:

[0036]

[0037] 2.2.4) Constraint 3: The sum of the computing resource overhead incurred by the edge server for providing computing tasks for different edge computing tasks should be less than the computing resource threshold on the edge computing server:

[0038]

[0039] 2.2.5) Constraint 4: Edge computing tasks can only be offloaded to edge servers that can provide services for this type of computing tasks:

[0040]

[0041] 2.2.6) Constraint 5: An edge computing task can only be offloaded to one edge server:

[0042]

[0043] 2.2.7) Constraint 6, Decision Variable Should be Boolean data:

[0044]

[0045] 2.2.8) Constraint 7, Decision Variable Should be Boolean data:

[0046]

[0047] 2.2.9) Constraint 8: The threshold for the computing resource occupancy rate on the edge server must be a positive number and must not exceed the server's maximum computing resource capacity:

[0048] TH resource ∈(0,100%] (9)

[0049] Step (3) specifically includes:

[0050] 3.1) Given a reinforcement learning agent, define the agent's goal as obtaining, through training, an edge task offloading scheduling policy that maximizes the expected cumulative discounted reward. In this method, define the state as the current task schedule. Define the agent's action as changing the edge computing task offloading scheduling policy to maximize the cumulative discounted reward. Define the reinforcement learning training environment as an input dataset containing edge network environment information and edge computing task offloading request information.

[0051] 3.2) In order to transform the minimization problem into a maximization problem, the reinforcement learning reward in this method is defined as the negative value of the end-to-end delay of the edge task, so that some maximization reward algorithms can be used to solve the minimization problem.

[0052] 3.3) Initialize the value function or policy. Before training the agent, you need to initialize its task offloading policy. These initial values ​​are usually random, and as training progresses, the agent will gradually improve these task offloading policies to achieve better rewards.

[0053] 3.4) The agent interacts with the environment, performs actions, and observes the environment’s feedback. At each step, the agent observes the current state, chooses an action, performs that action, and then receives a reward signal and the next state from the environment.

[0054] 3.5) Calculate rewards and update task scheduling strategy function Based on the reward signal provided by the environment, the agent will update its parameters.

[0055] 3.6) Iterative Training. The agent loops through steps 3.4) to 3.5), continuously interacting with and learning from the environment, and performing multiple rounds of iterative training. This iterative process continues N times (N is a fixed value that can be customized in this method).

[0056] The edge task offloading policy generated by the reinforcement learning agent is subject to the constraints described above. A "critic-only" reinforcement learning approach is employed, which trains only the "critic" network, eliminating the "actor" network used in traditional "actor-critic" algorithms. During each interaction with the environment, the "critic" network estimates the optimal action in the current environment. Under this agent-generated task offloading policy, the sum of the end-to-end delays of all edge tasks from different edge users is treated as a reward from the environment. The agent learns from these reward-action pairs to improve its scheduling strategy.

[0057] Step (4) specifically includes:

[0058] 4.1) The set of training environments Ω for training the reinforcement learning agent and the baseline policy π for generating the training curriculum Relu Enter the course learning module.

[0059] 4.2) The curriculum learning module selects the environment Ω from the input reinforcement learning agent training environment cur , used as a training environment for reinforcement learning agents.

[0060] 4.3) The reinforcement learning agent is trained in the training environment given by the curriculum learning module and generates a task offloading scheduling strategy π in the given training environment RL , which is given a training environment Ω cur The total end-to-end delay of task offloading generated in

[0061] 4.4) Baseline policy π for generating training coursesRelu In a given training environment Ω cur The total end-to-end delay of task offloading is

[0062] 4.5) Record the reinforcement learning method and baseline strategy in a given training environment Ω cur The difference in the total end-to-end delay of the task caused by the scheduling policy generated above

[0063] 4.6) Select a training environment Ω′ that has never been selected from the training environment set Ω input to the course learning module cur As Ω in step 4.2) cur , and repeat steps 4.2) to 4.6) until all training environments Ω in the training environment set Ω of the course learning module are input cur All are traversed by steps 4.2) to 4.5).

[0064] 4.7) Based on ΔT in different training environments cur Arrange different training environments Ω in descending order of size cur , get the sequence Ω of the sorted training environment SORT .

[0065] 4.8) Step S48: Reinforce the learning agent according to Ω SORT The training environment in the sequence is trained in sequence, and after completing the training, a trained reinforcement learning agent is obtained.

[0066] The second aspect of the present invention relates to an edge task unloading scheduling device based on network fault perspective, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement an edge task unloading scheduling method based on network fault perspective of the present invention.

[0067] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method of the present invention for edge task offloading scheduling based on network fault perspective is implemented.

[0068] The present invention provides an edge task offloading scheduling method based on network fault perspective, comprising: after an edge device requests to offload a computing task to an edge server, using network fault perspective to predict the communication delay of task offloading, and constructing a mathematical model for characterizing the end-to-end delay of edge task offloading. Thereafter, a trained reinforcement learning model method is used to offload and schedule tasks. In order to improve the task offloading scheduling effect of the reinforcement learning model in a complex dynamic network environment, the present invention uses a course learning method to generate a series of course environments to assist the training of reinforcement learning agents. The present invention accurately predicts the network communication delay, constructs a mathematical model of the end-to-end delay of edge computing tasks, and uses reinforcement learning to perform task scheduling and generate reinforcement learning training environment courses based on course learning, thereby more accurately characterizing the end-to-end delay of edge computing task offloading, so that edge task offloading requests can be better scheduled in a complex and changeable edge network environment to achieve a lower total task end-to-end delay.

[0069] Compared with the existing technology, the beneficial effect of the present invention is that in a complex and changeable edge network environment (such as scenarios where the topology links between edge network servers frequently change and the network communication delay frequently changes), the edge computing task offloading scheduling strategy calculated based on this method can achieve lower end-to-end delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 Schematic diagram of the application scenario of the method of the present invention.

[0072] Figure 2 The present invention is a flow chart of communication delay analysis during task offloading based on network fault perspective.

[0073] Figure 3 It is a schematic diagram of the reinforcement learning execution flow of the method of the present invention.

[0074] Figure 4 It is a flow chart of the course learning method of the present invention for generating a course environment for reinforcement learning agent training. DETAILED DESCRIPTION

[0075] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0076] It should be understood that the terms described herein are intended only to describe particular embodiments and are not intended to limit the present invention. In addition, for numerical ranges herein, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Each smaller range between any intermediate value within a stated value or stated range and any other stated value or intermediate value within the stated range is also encompassed by the present invention. The upper and lower limits of these smaller ranges may be independently included or excluded within the scope.

[0077] Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. Although only preferred methods and materials are described herein, any methods and materials similar or equivalent to those described herein may also be used in the practice or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods and / or materials associated with the documents. In the event of any conflict with any incorporated document, the contents of this specification shall prevail.

[0078] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments described herein without departing from the scope or spirit of the invention. Other embodiments will be apparent to those skilled in the art from the description of the invention. The present description and examples are intended to be illustrative only.

[0079] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0080] Unless otherwise specified, the "parts" described in the present invention are based on mass parts.

[0081] Example 1

[0082] The speech recognition method of the edge task offloading scheduling method based on network fault perspective provided by the present invention is applied, such as Figure 1 As shown in Figure 2, this method can quickly and accurately schedule offload requests for speech recognition tasks from different edge devices in complex edge network scenarios where edge network communication latency varies greatly and edge server network topology is constantly changing, thereby minimizing the total end-to-end latency of different speech recognition tasks. Figure 1 As shown, in the scenario defined by this method, resource-rich speech recognition servers are deployed near base stations and close to edge users so that nearby edge users can effectively perform speech recognition tasks. Base stations are interconnected via the Internet, and the latency and topology of the edge network vary over time. This method is designed as a modular server-side extension, and its specific implementation is as follows:

[0083] (1) Network delay analysis based on network fault perspective. Since the speech recognition task scheduling of this method needs to be based on the end-to-end delay of the speech recognition task (that is, the sum of the communication delay of the speech recognition task offloading and the computational delay of the speech recognition task generated by the recognition), the computational delay of the speech recognition task can be accurately predicted by the server based on historical experience. In a complex network environment, the analysis of communication delay is more difficult due to the fluctuation of network delay and the change of network topology. This method designs a responsive network fault perspective to analyze the communication delay of task offloading in a fluctuating network environment. Figure 2 As shown, the implementation method is as follows:

[0084] 1.1) When an edge user requests to offload the speech recognition task for the first time, this method probes the network to analyze the communication delay from the client to each server.

[0085] 1.2) When the edge user next requests to offload the speech recognition task, this method first proactively collects latency information from the K (in this method, set to 5 based on experience) nearest speech recognition servers and determines whether network transmission has caused a significant change in communication latency compared to the last time the speech recognition task was requested. This is because even if there are network fluctuations, these fluctuations are limited in a short period of time, and the latency to the nearest server is the best indicator of network fluctuations.

[0086] 1.3) If the relative order of the delay values ​​between the speech recognition servers changes significantly compared to the last time, or any delay value deviates from the threshold (set to 10% based on experience in this method), it is considered necessary to re-probe the edge speech recognition server network to analyze the communication delay when offloading the speech recognition task to different edge servers.

[0087] (2) Construction of mathematical model for end-to-end delay of speech recognition task. When different edge users initiate speech recognition task offloading requests, this method first constructs a mathematical model to characterize the end-to-end delay of speech recognition task offloading.

[0088] 2.1) The variables involved in the mathematical model constructed by this method are as follows:

[0089] 2.1.1) M, the number of edge speech recognition servers, is a positive integer and describes the number of servers that can be used to provide speech recognition services to different edge users in this method.

[0090] 2.1.2) N, the number of edge users, is a positive integer that describes the number of edge users that can initiate speech recognition task offloading requests to the edge server in this method.

[0091] 2.1.3)T n,k, which is the end-to-end task delay when edge user n requests to offload speech recognition task k to the edge speech recognition server. This is also the total computational delay incurred when the speech recognition task is offloaded to the edge speech recognition server. This delay is numerically equal to the communication delay during speech recognition task offloading and the computational delay on the edge server after the speech recognition task is offloaded to the edge speech recognition server.

[0092] 2.1.4)H n,k , which is the end-to-end task delay threshold when edge user n requests to offload speech recognition task k to the edge server, that is, the longest end-to-end delay that the speech recognition task can tolerate.

[0093] 2.1.5) The data type is Boolean and is used to describe whether the speech recognition task k from edge user n is offloaded to edge server m.

[0094] 2.1.6) The data type is Boolean and is used to describe whether the edge server m can provide computing services for the speech recognition task k.

[0095] 2.1.7) Describes the communication delay between edge server m and edge user n due to offloading of speech recognition tasks, obtained through network fault analysis.

[0096] 2.1.8) Used to describe the computational delay between edge server m and edge user n caused by offloading the speech recognition task, which is predicted by the server.

[0097] 2.1.9)c n,k , is used to describe the amount of computing resources requested by edge user n to the server providing speech recognition services when initiating an offloading request for speech recognition task k.

[0098] 2.1.10)TH resource ,The data type is percentage data, which is used to describe the threshold of computing resource occupancy rate on the edge server. In order to meet its own needs and some sudden computing task requirements, the edge computing server may need to reserve some computing resources. Therefore, when the current computing resource occupancy rate on the edge server is higher than the threshold TH resource When , the speech recognition task newly assigned to the edge server will wait to be served until the computing resource occupancy rate on the server drops below the threshold.

[0099] 2.1.11)C m , used to describe the total amount of computing resources on edge server m.

[0100] 2.2) The mathematical model constructed by this method to describe the end-to-end latency after speech recognition task offloading is as follows:

[0101] 2.2.1) The optimization goal of the task offloading scheduling strategy in this method is to minimize the sum of the end-to-end delays of all speech recognition tasks from edge users, that is:

[0102]

[0103] This method models the offloading scheduling problem of speech recognition tasks as a minimization problem, which is subject to the following constraints.

[0104] 2.2.2) Constraint 1: The end-to-end delay of offloading a speech recognition task k from edge user n to edge speech recognition server m is equal to the sum of the communication delay of offloading the task to the edge server and the computational delay of the task on the edge server:

[0105]

[0106] 2.2.3) Constraint 2: The end-to-end latency of speech recognition task k from edge user n when it is offloaded to edge server m should be lower than the latency threshold of the speech recognition task:

[0107]

[0108] 2.2.4) Constraint 3: The sum of the computing resource overhead incurred by the edge server for providing computations for different speech recognition tasks must be less than the computing resource threshold on the edge computing server:

[0109]

[0110] 2.2.5) Constraint 4: Speech recognition tasks can only be offloaded to edge servers that can provide services for this type of computing tasks:

[0111]

[0112] 2.2.6) Constraint 5: A speech recognition task can only be offloaded to one edge server:

[0113]

[0114] 2.2.7) Constraint 6, Decision Variable Should be Boolean data:

[0115]

[0116] 2.2.8) Constraint 7, Decision Variable Should be Boolean data:

[0117]

[0118] 2.2.9) Constraint 8: The threshold for the computing resource occupancy rate on the edge server must be a positive number and must not exceed the server's maximum computing resource capacity:

[0119] TH resource ∈(0,100%] (9)

[0120] (3) Speech recognition task offloading scheduling based on reinforcement learning. Since the solution of the above mathematical model is an NP-Hard problem, traditional solvers such as Gurobi will cause a large solution delay and cannot meet the needs of edge users to offload speech recognition tasks to edge servers in real time. Therefore, this method proposes a solution strategy based on reinforcement learning to improve the solution speed and maintain better solution accuracy in a dynamic network environment. Figure 3 As shown, the implementation method is as follows:

[0121] 3.1) Given a reinforcement learning agent, define the agent's goal as obtaining, through training, a speech recognition task offloading scheduling policy that maximizes the expected cumulative discounted reward. Define the state in this method to refer to the current task schedule. Define the agent's action as changing the speech recognition task offloading scheduling policy to maximize the cumulative discounted reward. Define the reinforcement learning training environment as an input dataset containing edge network environment information and speech recognition task offloading request information.

[0122] 3.2) In order to transform the minimization problem into a maximization problem, the reinforcement learning reward in this method is defined as the negative value of the end-to-end delay of the edge task, so that some maximization reward algorithms can be used to solve the minimization problem.

[0123] 3.3) Initialize the value function or policy. Before training the agent, you need to initialize its task offloading policy. These initial values ​​are usually random, and as training progresses, the agent will gradually improve these task offloading policies to achieve better rewards.

[0124] 3.4) The agent interacts with the environment, performs actions, and observes the environment’s feedback. At each step, the agent observes the current state, chooses an action, performs that action, and then receives a reward signal and the next state from the environment.

[0125] 3.5) Calculate rewards and update task scheduling strategy function Based on the reward signal provided by the environment, the agent will update its parameters.

[0126] 3.6) Iterative Training. The agent loops through steps 3.4) to 3.5), continuously interacting with and learning from the environment, and performing multiple rounds of iterative training. This iterative process continues N times (N is a fixed value that can be customized in this method).

[0127] In this method, the speech recognition task offloading strategy generated by the reinforcement learning agent must comply with the constraints described above. This method uses a "critic-only" reinforcement learning approach, which trains only the "critic" network, without the "actor" network of the traditional "actor-critic" algorithm. During each interaction with the environment, the "critic" network estimates the optimal action in the current environment. Under the agent-generated task offloading strategy, the sum of the end-to-end latency of all speech recognition tasks from different edge users is treated as a reward from the environment. The agent learns from the reward-action pairs to improve the agent's scheduling strategy.

[0128] (4) Reinforcement learning optimization strategy based on curriculum learning. When using reinforcement learning methods to generate speech recognition task offloading scheduling strategies, if the training environment of the agent spans a wide range of environments (for example, various possible edge network delay environments and various possible edge task types), the reinforcement learning strategy may perform poorly even if the agent's performance is tested in an environment with the same distribution as the training environment. This method uses curriculum learning methods to improve the generalization ability and progressive performance of reinforcement learning. Figure 4 As shown, the steps are as follows:

[0129] 4.1) The set of training environments Ω for training the reinforcement learning agent and the baseline policy π for generating the training curriculum Rule Enter the course learning module.

[0130] 4.2) The curriculum learning module selects the environment Ω from the input reinforcement learning agent training environment cur , used as a training environment for reinforcement learning agents.

[0131] 4.3) The reinforcement learning agent is trained in the training environment given by the curriculum learning module and generates a task offloading scheduling strategy π in the given training environment RL , which is given a training environment Ω cur The total end-to-end delay of task offloading generated in

[0132] 4.4) Baseline policy π for generating training courses Rule In a given training environment Ω cur The total end-to-end delay of task offloading is

[0133] 4.5) Record the reinforcement learning method and baseline strategy in a given training environment Ωcur The difference in the total end-to-end delay of the task caused by the scheduling policy generated above

[0134] 4.6) Select a training environment Ω′ that has never been selected from the training environment set Ω input to the course learning module cur As Ω in step S2 cur , and repeat steps 4.2) to 4.6) until all training environments Ω in the training environment set Ω of the course learning module are input cur All are traversed by steps 4.2) to 4.5).

[0135] 4.7) Based on ΔT in different training environments cur Arrange different training environments Ω in descending order of size cur , get the sequence Ω of the sorted training environment SORT .

[0136] 4.8) Reinforcement learning agent according to Ω SORT The training environment in the sequence is trained in sequence, and after completing the training, a trained reinforcement learning agent is obtained.

[0137] The reinforcement learning agent trained with the assistance of course learning can provide better speech recognition task offloading scheduling strategies in complex and dynamic network environments and can adapt to more complex network environments.

[0138] (5) According to the speech recognition task offloading scheduling strategy, speech recognition task offloading is realized.

[0139] When an edge user requests to offload a speech recognition task from an edge server, this method inputs the edge user's speech recognition task offloading request into the trained model, obtains a task offloading scheduling strategy, and then implements the offloading of the speech recognition task according to the above scheduling strategy.

[0140] Example 2

[0141] This embodiment relates to an edge task offloading scheduling system based on network fault perspective, including:

[0142] A network delay analysis module based on network fault perspective is used to collect and process network delay information when offloading speech recognition tasks for decision-making task scheduling strategies;

[0143] An end-to-end delay mathematical model construction module is used to construct a mathematical model representing the speech recognition task offloading requests of each edge user to minimize the sum of the end-to-end delays of different speech recognition tasks;

[0144] The task offloading scheduling module based on reinforcement learning is used to schedule edge computing task offloading requests from different edge users using the reinforcement learning model;

[0145] The curriculum-based reinforcement learning optimization module is used to generate environmental courses to assist reinforcement learning training and improve the robustness and versatility of reinforcement learning agents in dynamic network scenarios.

[0146] Example 3

[0147] This embodiment relates to an edge task offloading scheduling device based on network fault perspective, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement an edge task offloading scheduling method based on network fault perspective of embodiment 1.

[0148] Example 4

[0149] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the edge task offloading scheduling method based on network fault perspective of embodiment 1 is implemented.

[0150] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for scheduling edge task offloading based on network fault perspective, characterized in that: The following steps are included: (1) Network delay analysis based on network fault perspective: Task scheduling requires end-to-end delay based on tasks, where the computational delay of tasks is accurately predicted by the server based on historical experience. In complex network environments, communication delay analysis is difficult due to fluctuations in network delay and changes in network topology. Therefore, a responsive network fault perspective is designed to analyze the communication delay of task offloading in a fluctuating network environment. (2) Construct a mathematical model for the end-to-end delay of edge tasks; when different edge users initiate edge computing task offloading requests, construct a mathematical model that describes the end-to-end delay after edge task offloading; (3) Perform task offloading scheduling based on reinforcement learning; propose a solution strategy based on reinforcement learning for solving the mathematical model to improve the solution speed and maintain better solution accuracy in a dynamic network environment; (4) Reinforcement learning optimization strategy based on curriculum learning: When using reinforcement learning methods to generate edge task offloading scheduling strategies, if the training environment of the agent spans a wide range of environments, even if the agent's performance is tested in an environment with the same distribution as the training environment, the reinforcement learning strategy will perform poorly. To address this situation, a curriculum learning method is used to improve the generalization ability and asymptotic performance of reinforcement learning. Step (1) specifically includes: 1.1) When an edge user requests to offload a task for the first time, the network is probed to analyze the communication delay from the client to each server; 1.2) When the edge user requests to offload a task next time, it first proactively collects latency information from the K nearest servers and determines whether the communication latency has changed significantly due to network transmission compared to the last time it requested to offload a task; 1.3) If the relative order of the delay values ​​between servers changes significantly compared to the last time, or any delay value deviates from the threshold, it is considered necessary to re-probe the server network to analyze the communication delay when offloading edge tasks to different edge servers; Step (2) specifically includes: 2.1) The variables involved in the constructed mathematical model are as follows: 2.1.1) M, the number of edge servers, is a positive integer that describes the number of servers available to provide edge computing services to different edge users; 2.1.2) N, the number of edge users, is a positive integer that describes the number of edge users that can initiate edge task offloading requests to the edge server; 2.1.3)T n,k , which is the end-to-end task delay when edge user n requests to offload edge computing task k to the edge server, that is, the total delay caused by offloading the edge task to the edge server for calculation; its numerical value is equal to the communication delay during the edge task offloading process and the calculation delay of the edge computing task on the edge server after it is offloaded to the edge server; 2.1.4)H n,k , which is the end-to-end task delay threshold when edge user n requests to offload edge computing task k to the edge server, that is, the longest end-to-end delay that the edge task can tolerate; 2.1.5) The data type is Boolean, which is used to describe whether the edge computing task k from edge user n is offloaded to edge server m; 2.1.6) The data type is Boolean, which is used to describe whether the edge server m can provide computing services for the edge computing task k; 2.1.7) Used to describe the communication delay between edge server m and edge user n caused by offloading edge computing tasks, obtained from the perspective of network faults; 2.1.8) It is used to describe the computational delay between edge server m and edge user n caused by offloading edge computing tasks, which is predicted by the server; 2.1.9)c n,k , used to describe the amount of computing resources requested by edge user n to the server providing edge computing services when initiating an offloading request for edge computing task k; 2.1.10)TH resource ,The data type is percentage data, which is used to describe the threshold of computing resource occupancy rate on the edge server; the edge computing server needs to reserve some computing resources to meet its own needs and some sudden computing task needs, so when the current computing resource occupancy rate on the edge server is higher than the threshold TH resource When , the edge computing task newly assigned to the edge server will wait for computing services to be provided until the computing resource occupancy rate on the server drops below the threshold; 2.1.11)C m , used to describe the total amount of computing resources on the edge server m; 2.2) The mathematical model constructed to describe the end-to-end latency after edge computing task offloading is as follows: 2.2.1) The optimization goal of the task offloading scheduling strategy is to minimize the sum of the end-to-end delays of all edge computing tasks from edge users, that is: The offloading scheduling problem of edge computing tasks is modeled as a minimization problem, which is subject to the following constraints; 2.2.2) Constraint 1: The end-to-end delay of edge computing task k from edge user n being offloaded to edge server m is equal to the sum of the communication delay of the task being offloaded to the edge server and the computation delay of the task on the edge server: 2.2.3) Constraint 2: The end-to-end delay of edge computing task k from edge user n when it is offloaded to edge server m should be lower than the delay threshold of the edge computing task: 2.2.4) Constraint 3: The sum of the computing resource overhead incurred by the edge server for providing computing tasks for different edge computing tasks should be less than the computing resource threshold on the edge computing server: 2.2.5) Constraint 4: Edge computing tasks can only be offloaded to edge servers that can provide services for this type of computing tasks: 2.2.6) Constraint 5: An edge computing task can only be offloaded to one edge server: 2.2.7) Constraint 6, Decision Variable Should be Boolean data: 2.2.8) Constraint 7, Decision Variable Should be Boolean data: 2.2.9) Constraint 8: The threshold for the computing resource occupancy rate on the edge server must be a positive number and must not exceed the server's maximum computing resource capacity: TH resource ∈(0,100%] (9) Step (3) specifically includes: 3.1) Given a reinforcement learning agent, define the agent's goal as obtaining an edge task offloading scheduling policy that maximizes the expected cumulative discounted reward through training; define the state as the current task schedule; define the agent's action as changing the edge computing task offloading scheduling policy to maximize the cumulative discounted reward; define the reinforcement learning training environment as the input dataset containing edge network environment information and edge computing task offloading request information; 3.2) To transform the minimization problem into a maximization problem, the reinforcement learning reward is defined as the negative of the end-to-end delay of the edge task, so that some maximization reward algorithms can be used to solve the minimization problem; 3.3) Initialize the value function or policy: Before starting to train the agent, you need to initialize the agent's task offloading policy. These initial values ​​are usually random, and as training progresses, the agent will gradually improve these task offloading policies to obtain better rewards. 3.4) The agent interacts with the environment, performing actions and observing the environment's feedback; at each step, the agent observes the current state, chooses an action, performs that action, and then receives a reward signal and the next state from the environment; 3.5) Calculate rewards and update task scheduling policy functions. Based on the reward signal provided by the environment, the agent will update its parameters. 3.6) Iterative training: The agent cyclically executes steps 3.4) to 3.5), continuously interacting and learning with the environment, and performs multiple rounds of iterative training. This iterative process lasts N times.

2. The edge task offloading scheduling method based on network fault perspective according to claim 1, characterized in that: The edge task offloading strategy generated by the reinforcement learning agent is subject to the constraints mentioned above. It adopts a "critic-only" reinforcement learning method, which only trains the "critic" network, without the "actor" network in the traditional "actor-critic" algorithm. During each interaction with the environment, the "critic" network estimates the optimal action in the current environment. Under the agent-generated task offloading strategy, the sum of the end-to-end delays of all edge tasks from different edge users is treated as a reward from the environment. The agent learns from the reward-action pairs to improve the agent's scheduling strategy.

3. The edge task offloading scheduling method based on network fault perspective according to claim 1, characterized in that: Step (4) specifically includes: 4.1) The set of training environments Ω for training the reinforcement learning agent and the baseline policy π for generating the training curriculum Rule Enter the course learning module; 4.2) The curriculum learning module selects the environment Ω from the input reinforcement learning agent training environment cur , used as a training environment for reinforcement learning agents; 4.3) The reinforcement learning agent is trained in the training environment given by the curriculum learning module and generates a task offloading scheduling strategy π in the given training environment RL , which is given a training environment Ω cur The total end-to-end delay of task offloading generated in 4.4) Baseline policy π for generating training courses Rule In a given training environment Ω cur The total end-to-end delay of task offloading is 4.5) Record the reinforcement learning method and baseline strategy in a given training environment Ω cur The difference in the total end-to-end delay of the task caused by the scheduling policy generated above 4.6) Select a training environment Ω′ that has never been selected from the training environment set Ω input to the course learning module cur As Ω in step 4.2) cur , and repeat steps 4.2) to 4.6) until all training environments Ω in the training environment set Ω of the course learning module are input cur All are traversed from step 4.2) to step 4.5); 4.7) Based on ΔT in different training environments cur Arrange different training environments Ω in descending order of size cur , get the sequence Ω of the sorted training environment SORT ; 4.8) Reinforcement learning agent according to Ω SORT The training environment in the sequence is trained in sequence, and after completing the training, a trained reinforcement learning agent is obtained.

4. A device for scheduling edge task offloading based on network fault perspective, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement an edge task offloading scheduling method based on network fault perspective according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, an edge task offloading scheduling method based on network fault perspective according to any one of claims 1 to 3 is implemented.

Images