Task offloading method and computer equipment

Abstract

The present application discloses a task offloading method and computer equipment, which are applied to edge computing systems, including: obtaining a directed acyclic graph task of each terminal device, and determining the current subtask based on the directed acyclic graph task; for the current subtask of each terminal device, obtaining the state information of the edge computing system at the current moment, inputting the state information at the current moment into a multi-objective reinforcement learning model, generating a task offloading strategy corresponding to the current subtask, and based on the task offloading rate in the task offloading strategy corresponding to the current subtask of each terminal device, the terminal device and / or the server performs the corresponding task offloading operation on the current subtask. The present application can provide a task offloading strategy for the directed acyclic graph task of the terminal device, so as to obtain the optimal task offloading decision and computing resource allocation decision.

Description

Technical Field

[0001] The present application relates to the field of edge computing technology, and in particular to a task offloading method and computer device. Background Art

[0002] With the rapid development of the Internet of Things, the number of devices and the amount of data generated have increased dramatically, which has put forward higher requirements for real-time processing, big data analysis and device energy management. Terminal devices often face problems such as limited computing power, insufficient storage capacity and insufficient battery life when performing computationally intensive tasks. Therefore, edge computing solutions can effectively reduce latency and reduce the energy consumption of devices by offloading tasks from mobile devices to edge servers. However, edge computing also faces challenges such as limited resources and complex task scheduling in dynamic environments, and it is necessary to strike a balance between task completion time and device energy consumption. However, in actual applications, tasks are often related to each other. Different tasks and scenarios may cause the importance of each goal to change at any time, making it impossible to effectively capture the impact of task requirements or environmental changes, and lacking flexibility in resource allocation and task offloading decisions. Summary of the invention

[0003] The purpose of the present application is to provide a task offloading method that can provide a task offloading strategy for directed acyclic graph tasks of a terminal device so as to obtain the optimal task offloading decision and computing resource allocation decision.

[0004] Based on the above purpose, the present application provides a task offloading method, which is applied to an edge computing system, the edge computing system includes multiple terminal devices and multiple servers, and the method includes:

[0005] Obtain a directed acyclic graph task for each terminal device, and determine a current subtask based on the directed acyclic graph task, wherein the directed acyclic graph task includes a plurality of subtasks with dependency relationships;

[0006] For each current subtask of the terminal device, obtain the current state information of the edge computing system, input the current state information into the multi-objective reinforcement learning model, and generate the task offloading strategy corresponding to the current subtask;

[0007] Based on the task offloading rate in the task offloading policy corresponding to the current subtask of each terminal device, the terminal device and / or the server performs a corresponding task offloading operation on the current subtask.

[0008] Furthermore, the current state information is input into the multi-objective reinforcement learning model to generate the task offloading strategy corresponding to the current subtask, including:

[0009] The optimization model with the latency of directed acyclic graph tasks and the total energy consumption of terminal devices as the objectives is converted into a Markov decision process, and a multi-objective reinforcement learning model is constructed based on the Markov decision process;

[0010] The current state information and the current target weight vector are input into the multi-objective reinforcement learning model to generate a task offloading strategy corresponding to the current subtask, where the target weight vector includes a delay weight vector and an energy consumption weight vector; the task offloading strategy includes a task offloading rate, and the device that performs the task offloading and the allocated computing resources are determined based on the task offloading rate.

[0011] Furthermore, the optimization model with the latency and energy consumption of the directed acyclic graph task as the target is converted into a Markov decision process, and a multi-objective reinforcement learning model is constructed based on the Markov decision process. The multi-objective reinforcement learning model includes:

[0012] For each DAG task of a terminal device, a transmission network model of the DAG task is constructed based on the network architecture between the edge computing system and the terminal device. A delay model and a total energy consumption model of the DAG task are constructed based on the transmission network model. An optimization model with the delay of the DAG task and the total energy consumption of the terminal device as the target is constructed based on the delay model and the total energy consumption model.

[0013] A Markov decision process is established based on the optimization model, and the state space, mixed action space and reward function of the Markov decision process are set, wherein the state space is a collection of state information, and the state information includes the data size of the current subtask, the communication channel state, the available computing resources of the terminal device and the available computing resources of each server; the mixed action space includes a collection of discrete actions and a collection of continuous actions, and the discrete actions include: when the task offloading rate is a first value, all the current subtasks are offloaded to the terminal device, when the task offloading rate is a second value, all the current subtasks are offloaded to the target server, and when the task offloading rate is a value between the first value and the second value, a part of the current subtask is offloaded to the target server and another part of the current subtask is offloaded to the terminal device; the continuous action includes: the task offloading rate of the current subtask, and the computing resources allocated by the target server to the current subtask; the reward function includes a delay reward function and an energy consumption reward function;

[0014] The current state information and the current target weight vector are input into the multi-objective reinforcement learning model, and the multi-objective reinforcement learning model outputs the selected discrete actions and continuous actions, and the task offloading strategy corresponding to the current subtask is determined by the discrete actions and continuous actions.

[0015] Furthermore, a transmission network model of the directed acyclic graph task is constructed based on the network architecture between the edge computing system and the terminal device, and a delay model and a total energy consumption model of the directed acyclic graph task are constructed based on the transmission network model, including:

[0016] For each subtask in the directed acyclic graph task, a first transmission rate from the terminal device to the server and a second transmission rate from the server to the terminal device are determined based on Shannon's theorem, a transmission time of the subtask is determined based on the data size of the subtask, the data size of the calculation result, the first transmission rate and the second transmission rate, and a transmission energy consumption of the subtask in the terminal device is determined based on the transmission time, and a transmission network model is constructed based on the transmission time and transmission energy consumption of each subtask;

[0017] For each subtask in a directed acyclic graph task, a first task execution time of the subtask on the terminal device side is determined based on a task unloading rate of the subtask, a data size of the subtask, and resources allocated by the terminal device to the subtask; a second task execution time of the subtask on the server side is determined based on the task unloading rate of the subtask, the data size of the subtask, and resources allocated by the server to the subtask; the maximum of the first task execution time and the second task execution time is taken as the execution time of the subtask; the delay of the subtask is determined based on the transmission time and execution time of the subtask, and then the execution delay of the directed acyclic graph task is determined; a task delay model is constructed based on the execution delay of the directed acyclic graph task;

[0018] For each subtask in the directed acyclic graph task, the task execution energy consumption of the subtask in the terminal device is determined based on the first task execution time and the energy per unit time of the subtask, and the total energy consumption of the subtask is determined by the transmission energy consumption of the subtask in the terminal device and the task execution energy consumption of the subtask in the terminal device, and then the total energy consumption model of the terminal device is determined.

[0019] Furthermore, the multi-objective reinforcement learning model includes an experience replay buffer, an actor network and a critic network, the actor network includes discrete components and continuous components, the critic network includes two Q networks and two target Q networks, the Q network and the target Q network have the same structure, and the multi-objective reinforcement learning model is trained by using a distributed execution and centralized training method, including:

[0020] Distributed execution methods include:

[0021] Input the current state information and the current target weight vector into the discrete component and the continuous component in the actor network, the discrete component outputs the selected discrete action, and the continuous component outputs the selected continuous action;

[0022] End devices perform discrete and continuous actions to interact with the environment of the edge computing system;

[0023] Obtaining the state information at the next moment based on the interaction result, and determining the reward value obtained by the terminal device for executing the action based on the reward function;

[0024] The current state information, discrete actions, continuous actions, the current target weight vector, the next state information and the reward value constitute an experience sample and store it in the experience playback buffer;

[0025] Focused training methods include:

[0026] Using the experience samples in the experience replay buffer as training samples, the training samples are input into the critic network;

[0027] The two Q networks evaluate and output the corresponding actual Q values ​​based on the actions output by the actor network, and use the minimum value among the output actual Q as the minimum actual Q value output by the critic network, and update the parameters of the actor network with the minimum actual Q value;

[0028] The two target Q networks evaluate and output corresponding target Q values ​​based on the action output by the actor network at the next moment, and take the minimum value among the output target Q as the minimum target Q value. The parameters of the critic network are updated based on the minimum actual Q value and the minimum target Q value.

[0029] Furthermore, the parameters of the actor network are updated with the minimum actual Q value, including:

[0030] The discrete action loss function is determined by the weight coefficient of the discrete action and the probability of outputting the discrete action and the minimum actual Q value output by the critic network;

[0031] The continuous action loss function is determined by the weight coefficient of the continuous action, the probability of outputting the continuous action, and the minimum actual Q value output by the critic network;

[0032] Determine the first objective loss function of the actor network using the discrete action loss function and the continuous action loss function;

[0033] The model is trained based on the first objective loss function, a gradient descent process is performed on the first objective loss function, and the network parameters of the actor network are updated according to the result of the gradient descent process.

[0034] Furthermore, the parameters of the actor network are updated with the minimum actual Q value, including:

[0035] The discrete action loss function is determined by the weight coefficient of the discrete action and the probability of outputting the discrete action and the minimum actual Q value output by the critic network. for:

[0036] ;

[0037] in, Representing discrete actions The weight coefficient of It indicates the expectation of state s. Represents the state s and discrete actions The actor network outputs the probability of discrete actions, represents the minimum practical Q value, and Indicates parameters;

[0038] The continuous action loss function is determined by the weight coefficient of the continuous action and the probability of outputting the continuous action and the minimum actual Q value output by the critic network. for:

[0039] ;

[0040] in, Indicates continuous action The weight coefficient of It indicates the expectation of state s. Represents the state s and discrete actions The actor network outputs continuous actions The probability of Indicates the minimum practical Q value;

[0041] With discrete action loss function and the continuous action loss function Determine the first objective loss function of the actor network for:

[0042] = + ;

[0043] Model training is performed based on the first objective loss function, gradient descent processing is performed on the first objective loss function, and network parameters of the actor network are updated according to the result of the gradient descent processing.

[0044] Furthermore, the parameters of the critic network are updated based on the minimum actual Q value and the minimum target Q value, including:

[0045] The joint entropy of the strategy is determined by the probability of the actor network outputting discrete actions and the probability of outputting continuous actions, and the Bellman equation of the critic network evaluation function is updated based on the joint entropy;

[0046] The second objective loss function of the critic network is determined by the mean square error between the minimum actual Q value and the minimum target Q value, wherein the target Q value is obtained based on the following steps: inputting the state information of the next moment and the current weight vector into the actor network, the actor network outputs the action of the next moment, inputting the state information of the next moment, the action of the next moment and the current weight vector into the target Q network, obtaining the actual Q value of the next moment, and calculating the target Q value by the Bellman equation;

[0047] The model is trained based on the second objective loss function, the second loss function is subjected to gradient descent processing, and the network parameters of the critic network are updated according to the result of the gradient descent processing.

[0048] Furthermore, the critic network and the actor network are trained with sample information, including:

[0049] The joint entropy of the policy is determined by the probability of the actor network outputting discrete actions and the probability of outputting continuous actions. ;

[0050] ;

[0051] in, Representing discrete actions The weight coefficient of Represents the state s and discrete actions The actor network outputs discrete actions The probability of Indicates continuous action The weight coefficient of Represents the state s and discrete actions The actor network outputs continuous actions The probability of Indicates parameters;

[0052] The Bellman equation for updating the critic network Q function based on the joint entropy is:

[0053] ;

[0054] in, Indicates that the action is performed in state s The instant rewards you receive, represents the discount factor, Indicates selecting the smallest Q value from two Q networks, Indicates the state at the next moment. represents the discrete action at the next moment, Indicates the continuous action at the next moment;

[0055] The second objective loss function of the critic network is determined by the mean square error between the minimum actual Q value and the minimum target Q value, wherein the target Q value is obtained based on the following steps: inputting the state information of the next moment and the current weight vector into the actor network, the actor network outputs the action of the next moment, inputting the state information of the next moment, the action of the next moment and the current weight vector into the target Q network, obtaining the actual Q value of the next moment, calculating the target Q value through the Bellman equation, and the second objective loss function It is expressed as:

[0056] ;

[0057] in, represents the minimum target Q value, represents the minimum practical Q value, Indicates parameters;

[0058] The model is trained based on the second objective loss function, a gradient descent process is performed on the second objective loss function, and the network parameters of the critic network are updated according to the result of the gradient descent process.

[0059] Furthermore, the historical experience in the experience replay buffer is used as a training sample, including:

[0060] Sorting the experience samples in the experience replay buffer according to the priority sequence of the time difference error;

[0061] Select λb experience samples as the first experience sample set based on the sorted experience samples;

[0062] Based on the state error and weight error between adjacent experience samples, the similarity between adjacent experience samples is calculated to determine the similarity of each experience sample in the first experience sample set;

[0063] The time correlation of each experience sample in the first experience sample set is determined based on the time factor, where the time factor is set to , represents the exploration factor, T represents the current training time step, Indicates the time step number of the empirical sample sampling;

[0064] Determine a new priority for each experience sample based on the similarity and time correlation of each experience sample;

[0065] The first experience sample set is sorted in descending order based on the new priority, and the first b experience samples are selected from the sorted first experience sample set for sampling, and the sampled samples are used as training samples.

[0066] Based on the above purpose, the present application provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the task offloading method described above when executing the computer program.

[0067] This application obtains the state information of the environmental interaction between the terminal device and the edge computing system, determines the task offloading strategy of the current subtask of each terminal device based on the multi-objective reinforcement learning model, fully considers the connection between tasks, and determines the optimal task offloading strategy for the terminal device. It can more comprehensively evaluate the environmental interaction state of the terminal device and the edge server, the resource requirements of the task, and solve more complex task offloading problems, thereby improving the accuracy and efficiency of task offloading decisions, and can flexibly allocate various tasks between the terminal device and the edge server to reduce device load, reduce latency and save energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a schematic diagram of an edge computing system provided according to an embodiment of the present application;

[0069] Figure 2 is a first flow chart of a task offloading method provided according to an embodiment of the present application;

[0070] Figure 3 is a second flow chart of the task offloading method provided in an embodiment of the present application;

[0071] Figure 4 is a third flow chart of the task offloading method provided according to an embodiment of the present application;

[0072] Figure 5 is a fourth flow chart of the task offloading method provided according to an embodiment of the present application;

[0073] Figure 6 is a schematic diagram of the network structure of a multi-objective reinforcement learning model provided according to an embodiment of the present application;

[0074] Figure 7 is a fifth flow chart of the task offloading method provided according to an embodiment of the present application;

[0075] Figure 8 is a sixth flow chart of the task offloading method provided according to an embodiment of the present application;

[0076] Fig. 9 is a structural diagram of a task offloading device provided according to an embodiment of the present application;

[0077] Fig.10 It is a schematic diagram of the structure of a computer device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0078] The present application will be described in detail below in conjunction with the specific implementation modes shown in the accompanying drawings, but these implementation modes do not limit the present application. Structural, methodological, or functional changes made by ordinary technicians in the field based on these implementation modes are included in the protection scope of the present application.

[0079] MEC (Multi-access Edge Computing) is a multi-access edge computing that aims to place computing and storage resources at the edge nodes (servers) of the network, in order to be closer to users, so as to achieve lower latency, faster data processing and response capabilities. The edge server is a computing server that offloads the computing load on the terminal side to the edge of the network, providing cloud-like computing, storage and processing capabilities at the edge of the network. The embodiment of the present application is applied to an edge computing system, which includes multiple servers and multiple terminal devices.

[0080] like Figure 1 The schematic diagram of the edge computing system of the embodiment of the present application is shown in FIG. Figure 1 As shown in the figure, a task offloading scenario between a terminal device and an edge server in an edge computing environment is shown, wherein a directed acyclic graph task diagram of a terminal device 5 is shown. The user layer includes five terminal devices: terminal device 1, terminal device 2, terminal device 3, terminal device 4, and terminal device 5, and the edge layer includes edge servers: edge server 1, edge server 2, and edge server 3. The terminal devices can offload tasks to different servers for processing via wireless networks, and there is a competitive relationship between the terminal devices for wireless network resources and computing resources. The server can accept task requests from terminal devices and perform processing. The terminal device is connected to the server via wireless signals. As shown in the figure, Figure 1 The directed acyclic graph tasks of the terminal device 5 shown in the figure, the circular nodes in the task graph represent subtasks, and the arrows represent the dependencies between subtasks. The subtasks must be executed in the order of the arrows in the figure. Only after the predecessor task is completed can the subsequent task be executed. The task offloading method adopted in this application determines the optimal task offloading strategy of the terminal device through the dynamic state between the terminal device, the network and the server, such as network bandwidth, device computing power, task complexity, etc., and flexibly allocates tasks between resource-constrained local devices (such as smart phones or IoT devices) and edge servers to reduce device load, reduce latency and save energy consumption.

[0081] See also Figure 2 , Figure 2 A flow chart of a task offloading method provided in an embodiment of the present application is shown. The method is applied to an edge computing system and includes steps S201 to S203.

[0082] Step S201, obtaining a directed acyclic graph task of each terminal device, and determining a current subtask based on the directed acyclic graph task, wherein the directed acyclic graph task includes a plurality of subtasks having dependency relationships.

[0083] Each terminal device in the system randomly generates a directed acyclic graph task. Each node in the directed acyclic graph task corresponds to multiple subtasks one by one, and there are dependencies between multiple subtasks. The directed edge between each two nodes in the directed acyclic graph is used to represent the dependency between the corresponding two subtasks. Each task must be executed in the order of the dependency, and the subsequent task must wait for the completion of the previous task before it can be processed. For each terminal device, the current subtask at the current moment is determined based on the directed acyclic graph task of the terminal device.

[0084] Exemplarily, the directed acyclic graph task may be a machine learning computing task, such as a video surveillance analysis task, a traffic management computing task, and the like.

[0085] Step S202, for the current subtask of each terminal device, obtain the current state information of the edge computing system, input the current state information into the multi-objective reinforcement learning model, and generate a task offloading strategy corresponding to the current subtask.

[0086] Each terminal device interacts with the environment of the edge computing system in real time, obtains the current state information, inputs the current state information into the multi-objective reinforcement learning model for processing, and outputs the task offloading strategy corresponding to the current subtask. Based on the processing of the multi-objective reinforcement learning model, the task offloading strategy of the current subtask of each terminal device can be output. The task offloading strategy includes the task offloading rate. Based on the task offloading rate, the offloading strategy and resource allocation strategy of the current subtask can be determined to optimize the energy consumption and task execution time of the terminal device, so that each task can be flexibly allocated between the terminal device and the edge server to reduce the device load, reduce latency and save energy consumption.

[0087] Exemplarily, the status information includes the data size of the subtask, the data size of the result of the server calculating the subtask, the channel status of the terminal device and the server, the available computing resources of the terminal device, and the available computing resources of each server. The channel status can be the available network bandwidth between the terminal device and the server.

[0088] Step S203: based on the task offloading rate in the task offloading policy corresponding to the current subtask of each terminal device, the terminal device and / or the server performs a corresponding task offloading operation on the current subtask.

[0089] An uninstall rate in an uninstall policy corresponding to a current subtask of the terminal device is determined, and based on the uninstall rate, the terminal device and / or the server performs a corresponding task uninstall operation on the current subtask.

[0090] This implementation is based on the state information of the environmental interaction between the terminal device and the edge computing system, and determines the task offloading strategy of the current subtask of each terminal device through a multi-objective reinforcement learning algorithm. Each terminal device and / or each server performs the task offloading operation of the current subtask, fully considering the connection between tasks. The optimization goal of the task is fully considered in the reinforcement learning process to obtain the optimal offloading decision and resource allocation, and can more comprehensively evaluate the interaction state of the terminal device and the edge server environment, the resource requirements of the task, and solve more complex task offloading problems, thereby improving the accuracy and efficiency of task offloading decisions.

[0091] In some embodiments, Figure 3 As shown, the current state information is input into the multi-objective reinforcement learning model, and the steps of generating the task offloading strategy corresponding to the current subtask include:

[0092] S301, converting the optimization model with the latency of the directed acyclic graph task and the total energy consumption of the terminal device as the target into a Markov decision process, and constructing the multi-objective reinforcement learning model based on the Markov decision process;

[0093] Step S302, input the current state information and the current target weight vector into the multi-objective reinforcement learning model to generate a task offloading strategy corresponding to the current subtask, wherein the target weight vector includes a delay weight vector and an energy consumption weight vector; the task offloading strategy includes a task offloading rate, and the device for performing task offloading and the allocated computing resources are determined based on the task offloading rate.

[0094] Construct an optimization model with the latency of directed acyclic graph tasks and the total energy consumption of terminal devices as the target, and convert the optimization model into a Markov decision process, that is, convert the optimization process of the task offloading strategy into a Markov decision process, with the latency of the task and the total energy consumption of the terminal device as the target, and solve the optimal strategy of the Markov decision process through a multi-objective reinforcement learning algorithm. The weight vector of the target is input into the model as a parameter during the model training process, that is, the state information at the current moment, the delay weight vector at the current moment, and the energy consumption weight vector at the current moment are input into the multi-objective reinforcement learning model. The multi-objective reinforcement learning model determines the task offloading strategy corresponding to the current subtask based on the state information at the current moment, the delay weight vector at the current moment, and the energy consumption weight vector at the current moment. Exemplarily, the target weight vector at the current moment can be randomly obtained from a preference space (Preference Set).

[0095] In this implementation, a multi-objective reinforcement learning algorithm is used to solve the Markov decision process, with task delay and device energy consumption as targets, and the weight vector of the target is introduced. The task offloading strategy can be dynamically adjusted according to different target weights, and a trade-off can be made between multiple targets to cope with the task offloading strategy under different weight configurations. Users can choose the appropriate weight vector combination according to specific needs, which can achieve a better balance between task completion time and device energy consumption, obtain the best resource allocation and offloading decision, and improve the accuracy of task offloading decisions.

[0096] In some embodiments, Figure 4 As shown in FIG. 1 , the optimization model with the delay and energy consumption of the directed acyclic graph task as the target is converted into a Markov decision process, and a multi-objective reinforcement learning model is constructed based on the Markov decision process. The steps of the multi-objective reinforcement learning model include:

[0097] Step S401, for each directed acyclic graph task of the terminal device, a transmission network model of the directed acyclic graph task is constructed based on the network architecture between the edge computing system and the terminal device, a delay model and a total energy consumption model of the directed acyclic graph task are constructed based on the transmission network model, and an optimization model with the delay of the directed acyclic graph task and the total energy consumption of the terminal device as the target is constructed according to the delay model and the total energy consumption model.

[0098] For each directed acyclic graph task, based on the delay model and the total energy consumption model, an optimization model with task delay and total energy consumption of terminal equipment as optimization objectives is determined.

[0099] Step S402, establishing a Markov decision process based on the optimization model, and setting the state space, mixed action space and reward function of the Markov decision process.

[0100] The optimization model is transformed into a Markov decision process, and the state space, mixed action space and reward function of the Markov decision process are defined.

[0101] Exemplarily, the state space is a collection of state information, where the state information includes the data size of the current subtask, the communication channel state, the available computing resources of the terminal device, and the available computing resources of each server.

[0102] Exemplarily, the hybrid action space includes a set of discrete actions and a set of continuous actions. Discrete actions include: when the task offloading rate is a first value, all current subtasks are offloaded to the terminal device; when the task offloading rate is a second value, all current subtasks are offloaded to the target server; when the task offloading rate is a value between the first value and the second value, part of the current subtask is offloaded to the target server and the other part of the current subtask is offloaded to the terminal device. The first value and the second value are both integers, for example, the first value is 0 and the second value is 1. Continuous actions include: the task offloading rate of the current subtask, and the computing resources allocated by the target server to the current subtask. The target server is the server that the multi-objective reinforcement learning model allocates to the current subtask for task offloading based on the state information at the current moment.

[0103] Exemplarily, when the task offloading rate is 0, all current subtasks are offloaded to the terminal device, and the computing resources allocated to the current subtask by the target server are 0; when the task offloading rate is 1, all current subtasks are offloaded to the target server, and the computing resources allocated to the current subtask by the target server are determined; when the task offloading rate is a value between 0 and 1, part of the current subtask is offloaded to the target server and another part of the current subtask is offloaded to the terminal device, and the computing resources allocated to the current subtask by the target server are determined.

[0104] Exemplarily, the reward function includes a delay reward function and an energy consumption reward function. After the terminal device executes the task offloading strategy, the edge computing system determines the feedback reward through the reward function to motivate the model to output a more accurate task offloading strategy.

[0105] Step S403, input the current state information and the current target weight vector into the multi-objective reinforcement learning model, and the multi-objective reinforcement learning model outputs the selected discrete actions and continuous actions, and determines the task offloading strategy corresponding to the current subtask with the discrete actions and continuous actions.

[0106] The state information at the current moment in the state space and the target weight vector at the current moment randomly selected from the preference space are input into the multi-objective reinforcement learning model. Based on the state information at the current moment and the target weight vector at the current moment, the multi-objective reinforcement learning model selects the corresponding action from the mixed action space as output, that is, outputs discrete actions and continuous actions, and determines the task offloading strategy corresponding to the current subtask.

[0107] In some embodiments, based on the unloading rate in the unloading policy corresponding to the current subtask, the terminal device and / or the server performs the corresponding task unloading operation on the current subtask. For example, if the task unloading rate is 0, the terminal device performs the task unloading operation locally, and the computing resources allocated to the current subtask by the target server are 0; if the task unloading rate is 1, the target server performs the task unloading operation, and when the task unloading rate is a value between 0 and 1, based on the corresponding task unloading rate, the terminal device performs the task unloading of the corresponding part of the current subtask locally, and the server performs the task unloading of the corresponding part of the current subtask.

[0108] In this embodiment, an optimization model with task latency and total energy consumption of terminal equipment as targets is constructed, and the optimization model is converted into a Markov decision process. Based on discrete actions and continuous actions in the mixed action space of the Markov decision process, the discrete actions and continuous actions are combined to determine the task offloading strategy of the current subtask, so as to solve more complex task offloading problems and output a more optimized task offloading strategy.

[0109] In some embodiments, Figure 5 As shown, a transmission network model of a directed acyclic graph task is constructed based on the network architecture between the edge computing system and the terminal setting, and a delay model and a total energy consumption model of the directed acyclic graph task are constructed based on the task model and the transmission network model, including the following steps:

[0110] Step S501, for each subtask in the directed acyclic graph task, determine the first transmission rate from the terminal device to the server and the second transmission rate from the server to the terminal device based on Shannon's theorem, determine the transmission time of the subtask based on the data size of the subtask, the data size of the task result, the first transmission rate and the second transmission rate, determine the transmission energy consumption of the subtask in the terminal device based on the transmission time, and construct a transmission network model based on the transmission time and transmission energy consumption of each subtask.

[0111] For each directed acyclic graph task of each terminal device, the constituent elements and unloading strategies of each subtask in the directed acyclic graph task are defined, wherein the constituent elements include the data size of the subtask, the data size of the task result, the task unloading rate, the computing resources allocated by the terminal device to the subtask, and the computing resources allocated by the server to the subtask. The unloading strategy includes the inability to perform task unloading, the execution of task unloading at the terminal device, and the identification of the server that performs task unloading.

[0112] Exemplarily, the directed acyclic graph task set generated by all terminal devices is represented as , each DAG task can be represented as a series of subtasks Combination of , the connection between each subtask is expressed as , where the dependency is expressed as ,For example Represents a subtask and subtasks There are upstream and downstream dependencies between them, and the components of each subtask are defined as , Indicates the data size of the subtask, Indicates the data size of the task result of calculating this subtask. represents the task offloading rate, Indicates the computing resources of the subtask allocated to the terminal device n, Indicates the computing resources allocated to the subtasks on server m. The value range of the task offloading rate is [0,1]. The computing resources allocated to all current subtasks on a server must be less than the computing resources currently available on the server. Task offloading is performed in each time slot. If the subtask in the directed acyclic graph task is still running, it will not be unloaded. Therefore, the subtask offloading strategy can be expressed as , indicating that "-1" means that the subtask is still running and cannot be uninstalled, "0" means that it is processed by the local terminal device, and "1, 2, 3...M" indicates the server ID of the task uninstallation.

[0113] Exemplarily, when multiple terminal devices are unloading at the same time, the first transmission rate between the terminal device n and the server m is calculated based on the Shannon theorem. It can be expressed as:

[0114] ;

[0115] Among them, SNR represents the wireless network signal-to-noise ratio, N m It means that N terminal devices simultaneously unload tasks to server m in the current time step t. represents the available bandwidth of server m;

[0116] The server m transmits the task result back to the terminal device n at a second transmission rate for:

[0117] ;

[0118] in, Represents the available bandwidth of terminal device n.

[0119] Subtasks Transmission time for:

[0120] ;

[0121] ;

[0122] ;

[0123] in, Represents a subtask The data size, Represents a subtask The task offloading rate is Indicates that the server returns a subtask The data size of the calculation result;

[0124] Transmission energy consumption of subtasks in terminal devices It is expressed as:

[0125] ;

[0126] in, Represents the wireless transmission energy consumption per unit time.

[0127] Step S502: for each subtask in the directed acyclic graph task, the first task execution time of the subtask on the terminal device side is determined based on the task offloading rate, the data size of the subtask, and the resources allocated by the terminal device to the subtask; the second task execution time of the subtask on the server side is determined based on the task offloading rate, the data size of the subtask, and the resources allocated by the server to the subtask; the maximum of the first task execution time and the second task execution time is used as the execution time of the subtask; the delay of the subtask is determined based on the transmission time and the execution time of the subtask, and then the execution delay of the directed acyclic graph task is determined; and a task delay model is constructed based on the execution delay of the directed acyclic graph task.

[0128] Exemplarily, the execution time of the subtask is set to the maximum value of the execution time on the terminal device side and the execution time on the server side.

[0129] Subtasks Execution time It is expressed as:

[0130] ;

[0131] in, Represented as a subtask The first task execution time on the terminal device side, Represented as a subtask The second task execution time on the server side, Indicates that the terminal device n allocates subtasks of computing resources, Indicates that server m allocates subtasks computing resources;

[0132] Each time step will be used to record the completion time of the task, so the transmission time and execution time Need to be converted to time steps and then rounded up, subtask Delay for:

[0133] ;

[0134] in, represents the time interval of the time step;

[0135] The latency of a directed acyclic graph task is expressed as:

[0136] ;

[0137] ;

[0138] Among them, ST represents the time step of each subtask in the directed acyclic graph task, FT represents the time step of completion, and the FT of the last subtask is the delay of the entire directed acyclic graph task completion. express.

[0139] The latency T of the DAG task n It is expressed as:

[0140] .

[0141] Step S503, for each subtask in the directed acyclic graph task, determine the task execution energy consumption of the subtask in the terminal device based on the first task execution time and the energy per unit time of the subtask, determine the total energy consumption of the subtask by the transmission energy consumption of the subtask in the terminal device and the task execution energy consumption of the subtask in the terminal device, and then determine the total energy consumption model of the terminal device.

[0142] For example, the energy consumed by executing a subtask is the execution time of the terminal device multiplied by the energy per unit time, and the energy consumed by the server is negligible, so the subtask Energy consumption of task execution in terminal equipment It is expressed as:

[0143] ;

[0144] Where en represents the energy per unit time.

[0145] Subtasks The total energy consumption is expressed as for:

[0146] ;

[0147] Total energy consumption of terminal equipment E n It is expressed as:

[0148] .

[0149] According to the delay model and the total energy consumption model, the optimization model with delay and energy consumption as the optimization objectives is expressed as , that is, minimizing latency and energy consumption is taken as the optimization goal.

[0150] For example, the above optimization model is transformed into a Markov decision process, which is represented by six elements: , respectively represent the state space, action space, state transition probability, reward function, preference space and strategy set. In the preference space, each preference vector is described as , represents the delay weight vector, represents the energy consumption weight vector, The value range of is [0,1], Each subsequent training step is a Markov decision process.

[0151] The state space is the environmental information observed by the terminal device regarding the interaction between the terminal device and the edge system environment. The state space at time t is It is expressed as:

[0152] ;

[0153] in, Indicates the data size of the current subtask, Indicates the data size of the task result calculated by the current subtask. Indicates the current communication channel status, represents the currently available computing resources of terminal device n, represents the currently available computing resources of server m;

[0154] The hybrid action space is used to reflect the combination of actions that the terminal device can take. The hybrid action space is defined as:

[0155] ;

[0156] in, represents discrete actions, and Indicates continuous action. Indicates the task offloading rate of the current subtask, Indicates the computing resources allocated by the target server to the current subtask.

[0157] The reward function refers to the corresponding reward obtained based on the state of the action executed at each time step. Including delayed reward function and energy consumption reward function , respectively expressed as:

[0158] ;

[0159] ;

[0160] ;

[0161] ;

[0162] ;

[0163] Assuming that the task is not uninstalled, the time required for local processing on the terminal device is used as a benchmark to measure the reward after uninstallation. The reward can be positive, negative, or 0. For tasks that are still running and cannot be uninstalled, the desired offloading strategy is to process them locally on the terminal device. For example, 0.5 or -0.5 can be used as a reward to encourage the terminal device to make the correct offloading action.

[0164] In some embodiments, the multi-objective reinforcement learning model includes an experience replay buffer, an actor network and a critic network, the actor network includes a discrete component and a continuous component, the critic network includes two Q networks and two target Q networks, the Q network and the target Q network have the same structure, and the multi-objective reinforcement learning model is trained by combining distributed execution with centralized training, including:

[0165] Distributed execution methods include:

[0166] The state information at the current moment and the target weight vector at the current moment are input into the discrete components and continuous components in the actor network, the discrete components output the selected discrete actions and continuous actions, and the continuous components output the selected continuous actions; the terminal device executes discrete actions and continuous actions to interact with the environment of the edge computing system; the state information at the next moment is obtained based on the interaction result, and the reward value obtained by the terminal device when executing the action is determined based on the reward function; the state information at the current moment, the discrete actions, the continuous actions, the weight vector at the current moment, the state information at the next moment, and the reward value constitute an experience sample and store it in the experience replay buffer.

[0167] The centralized training method includes: using experience samples in the experience playback buffer as training samples, and inputting the training samples into the critic network; two Q networks respectively evaluate and output corresponding actual Q values ​​based on the actions output by the actor network, and use the minimum value among the output actual Q as the minimum actual Q value output by the critic network, and update the parameters of the actor network with the minimum actual Q value; two target Q networks respectively evaluate and output corresponding target Q values ​​based on the actions output by the actor network at the next moment, and use the minimum value among the output target Q as the minimum target Q value, and update the parameters of the critic network based on the minimum actual Q value and the minimum target Q value.

[0168] Exemplarily, based on the completion status of the current subtasks of each terminal device, the status of all current tasks is updated, the available computing resources of the terminal device is updated, and the network resources are updated, that is, the data size of the subtask at the next moment, the available computing resources of the terminal device, the available computing resources of the server, and the communication channel status are obtained.

[0169] Exemplarily, after the terminal device and / or the server performs the corresponding task offloading operation, the transmission time of the current subtask is calculated based on the transmission network model, the total energy consumption of the terminal device is calculated based on the total energy consumption model, and the delay of the current subtask is calculated based on the delay model. Based on the delay reward function and the delay of the current subtask, and based on the energy consumption reward function and the total energy consumption of the terminal device, the reward value obtained by the terminal device performing the action is determined.

[0170] In this implementation, a multi-objective reinforcement learning model is constructed based on a deep reinforcement learning model of the SAC (soft Actor-Critic) algorithm architecture. Figure 6The network architecture diagram shown in Figure 1 is a diagram of the network architecture. The multi-objective reinforcement learning algorithm aims to learn multiple objectives simultaneously to find the Pareto optimal solution set between multiple conflicting objectives. The multi-objective reinforcement learning model includes an experience replay buffer, an actor network, and a critic network. The actor network includes discrete components and continuous components. The experience replay buffer records the experience samples generated by the interaction between the terminal device and the environment. The actor network includes discrete components and continuous components, so a hybrid discrete and continuous action encoding is used in the actor network to achieve precise processing of action selection. Both the discrete component and the continuous component contain two fully connected neural networks, and the fully connected neural networks share hidden layers so that the two networks can jointly learn state representations, allowing information to be exchanged between the two network components, thereby better coordinating the selection of discrete and continuous actions. The actor network outputs the selected action based on the input state information and weight vector, namely, discrete actions and continuous actions. The discrete action determines which server the task is completely offloaded to, or all offloaded to the terminal device for processing, or partially offloaded to the terminal device and partially offloaded to the server. The continuous action determines the task offloading rate of the task offloading and the computing resources allocated to the task by the target server.

[0171] The critic network includes four Q networks, two Q networks and two target Q networks. The Q network and the target Q network have the same structure. Using two Q networks with the same structure to perform value estimation can reduce the probability of overestimation and reduce the instability introduced by estimation errors. The critic network outputs the action-state value for a given state, the action performed, and the weight vector, which is used to evaluate the value of the action taken by the actor network when the state and weight vector are given.

[0172] This implementation method is based on the neural network design of the multi-objective reinforcement learning algorithm to construct the network architecture of the multi-objective reinforcement learning model. The multi-objective reinforcement learning algorithm aims to learn multiple objectives at the same time to find the Pareto optimal solution set between multiple conflicting objectives. By using concave functions to enhance the objectives, the multi-objective space can be better explored and diversified Pareto optimal solutions can be found. The weight vector of the objectives is set to adjust the relative importance of each objective during the reinforcement learning process, and the Q value of the multiple objectives is scalarized using the weight vector of the objectives to guide the optimization of the task offloading strategy of the terminal device to meet the weight preferences of multiple objectives, and the task offloading strategy can be adjusted dynamically.

[0173] For example, Figure 6 As shown, in the distributed execution, the current state information and the weight vector at the current moment In the input actor network, the weight vector at the current moment includes the delay weight vector at the current moment and the energy consumption weight vector at the current moment. The discrete component receives the current state information Generating Discrete Action Distributions , the discrete action distribution is sampled to obtain discrete actions The continuous component receives the current state information , outputs continuous parameters related to continuous actions. Continuous parameters are usually the mean x̄ and standard deviation of the multivariate Gaussian distribution in the action space. , sample to get continuous action , that is, the actor network outputs the discrete action at the current moment and continuous action , that is, the current action space The terminal device interacts with the environment of the edge computing system based on the output discrete actions and continuous actions to obtain the state information of the next moment. , and the reward value fed back by the environment after executing the current action , the reward value is determined based on the above reward function. , current action space , the weight vector at the current moment , the status information of the next moment And the reward value Constructing an experience sample The experience replay buffer is stored as training samples to train the network. During centralized training, a batch of experience samples are extracted from the experience replay buffer to obtain training samples. , the critic network is trained with training samples, and the critic network is used to evaluate the actions selected by the actor network. The two output Q networks output two actual Q values, and the minimum actual Q value is selected and output to the actor network to update the actor network. The actor network continuously modifies the strategy according to the minimum actual Q value, thereby learning a better task offloading strategy.

[0174] In some embodiments, Figure 7 As shown, updating the parameters of the actor network with the minimum actual Q value comprises the steps of:

[0175] Step S701, determining a discrete action loss function using a weight coefficient of the discrete action, a probability of outputting the discrete action, and a minimum actual Q value output by the critic network;

[0176] Step S702, determining a continuous action loss function using the weight coefficient of the continuous action and the probability of outputting the continuous action and the minimum actual Q value output by the critic network;

[0177] Step S703, determining a first target loss function of the actor network using the discrete action loss function and the continuous action loss function;

[0178] Step S704: Perform model training based on the first objective loss function, perform gradient descent processing on the first objective loss function, and update the network parameters of the actor network according to the result of the gradient descent processing.

[0179] In this embodiment, the loss function of the actor network comprehensively considers the discrete action loss function and the continuous action loss function, and optimizes the discrete components and the continuous components at the same time, so that the network can be more adaptable to the strategy optimization of the discrete-continuous mixed action space. The use of the minimum actual Q value can ensure that the actor network is optimized in the direction of maximizing the Q value, thereby improving the accuracy of the network strategy prediction, thereby obtaining a strategy for better predicting task offloading.

[0180] For example, the discrete action loss function is determined by the weight coefficient of the discrete action and the probability of outputting the discrete action and the minimum actual Q value output by the critic network. for:

[0181] ;

[0182] in, Representing discrete actions The weight coefficient controls the contribution of discrete actions to the loss. It indicates the expectation of state s. represents the probability of the actor network outputting discrete actions. The logarithm is taken to maximize the probability, similar to the maximum likelihood estimation. represents the minimum actual Q value, represents the minimum of two actual Q values, and Indicates parameters;

[0183] The continuous action loss function is determined by the weight coefficient of the continuous action and the probability of outputting the continuous action and the minimum actual Q value output by the critic network. for:

[0184] ;

[0185] in, Indicates continuous action The weight coefficient of It indicates the expectation of state s. Represents the state s and discrete actions The actor network outputs the probability of consecutive actions, Indicates the minimum practical Q value;

[0186] With discrete action loss function and the continuous action loss function Determine the first objective loss function of the actor network for:

[0187] = + ;

[0188] The model is trained based on the first objective loss function, a gradient descent process is performed on the first objective loss function, and the network parameters of the actor network are updated according to the result of the gradient descent process.

[0189] In some embodiments, Figure 8 As shown, the parameters of the critic network are updated based on the minimum actual Q value and the minimum target Q value, including the steps of:

[0190] Step S801, determining the joint entropy of the strategy based on the probability of discrete actions and the probability of continuous actions output by the actor network, and updating the Bellman equation of the critic network Q function based on the joint entropy;

[0191] Step S802, determining a second objective loss function of the critic network by a mean square error between the minimum actual Q value and the minimum target Q value, wherein the target Q value is obtained based on the following steps: inputting the state information of the next moment and the current weight vector into the actor network, the actor network outputting the action of the next moment, inputting the state information of the next moment, the action of the next moment and the current weight vector into the target Q network, obtaining the actual Q value of the next moment, and calculating the target Q value by the Bellman equation;

[0192] Step S803, performing model training based on the second objective loss function, performing gradient descent processing on the second objective loss function, and updating the network parameters of the critic network according to the result of the gradient descent processing.

[0193] The SAC algorithm combines the principles of maximum entropy reinforcement learning and aims to optimize an objective function that takes into account the reward mechanism and policy entropy. SAC combines the Bellman equation with maximum entropy. The Bellman equation of the Q function is modified to include an entropy term, and the temperature parameter is used to control the importance of entropy. In the SAC algorithm, the strategy is updated by optimizing the KL divergence so that it can obtain high rewards and maintain high entropy. In this embodiment, based on the mixed action space of discrete and continuous, the entropy in the SAC algorithm is improved to adapt to discrete actions. The improved joint entropy combines discrete actions and continuous actions. The improved joint entropy is introduced into the Bellman equation for calculating the critic network loss function, so that the network can further improve the prediction of the task offloading strategy that combines discrete actions and continuous actions, thereby being able to obtain a strategy that better predicts task offloading.

[0194] For example, the joint entropy of the policy is determined by the probability of discrete actions and the probability of continuous actions output by the actor network ;

[0195] ;

[0196] in, Representing discrete actions The weight coefficient of represents the probability of the actor network outputting a discrete action, Indicates continuous action The weight coefficient of Represents the state s and discrete actions The actor network outputs the probability of consecutive actions, Indicates parameters;

[0197] The Bellman equation for updating the critic network Q function based on joint entropy is:

[0198] ;

[0199] in, Indicates that an action is performed in state s The instant rewards you receive, Represents the discount factor, ranging from [0, 1], which is used to measure the impact of future rewards on current decisions. The closer it is to 1, the more importance is placed on future rewards; Indicates selecting the smallest Q value from two Q networks, Indicates the next state, represents the next discrete action, Indicates the next continuous action;

[0200] The second objective loss function of the critic network is determined by the mean square error between the minimum actual Q value and the minimum target Q value, where the target Q value is obtained based on the following steps: the state information of the next moment and the current weight vector are input into the actor network, the actor network outputs the action of the next moment, the state information of the next moment, the action of the next moment and the current weight vector are input into the target Q network, the actual Q value of the next moment is obtained, and the target Q value is calculated by the Bellman equation. The second objective loss function It is expressed as:

[0201] ;

[0202] in, represents the minimum target Q value, represents the minimum practical Q value, The loss function of the critic network is determined by the mean squared error between the minimum actual Q value and the minimum target Q value, while the Q network gradually improves its estimate to make it more accurately reflect the action value.

[0203] In some embodiments, using historical experience in the experience replay buffer as a training sample includes the steps of:

[0204] Sorting the experience samples in the experience replay buffer according to the priority sequence of the time difference error;

[0205] Select λb experience samples as the first experience sample set based on the sorted experience samples;

[0206] Based on the state error and weight error between adjacent experience samples, the similarity between adjacent experience samples is calculated to determine the similarity of each experience sample in the first experience sample set;

[0207] The time correlation of each experience sample in the first experience sample set is determined based on the time factor, where the time factor is set to , represents the exploration factor, T represents the current training time step, Indicates the time step number of the empirical sample sampling;

[0208] Determine a new priority for each experience sample based on the similarity and time correlation of each experience sample;

[0209] The first experience sample set is sorted in descending order based on the new priority, and the first b experience samples are selected from the sorted first experience sample set for sampling, and the sampled samples are used as training samples.

[0210] In a dynamic multi-target environment, if random sampling is used to obtain samples in the experience replay buffer and the critic network is trained with the samples, extrapolation errors may be introduced, that is, the weight vectors and environmental states of the actor network and the critic network in the current training step are too different, and additional errors are introduced in the loss function calculation, so that the training effect is poor. In an embodiment of the present application, by sampling in the experience buffer based on the weight and state similarity ranking of the current training step, the actor network and the current critic network are kept consistent, which solves the technical problem of extrapolation errors and is closely integrated with the strategy experience. A time factor is introduced in the sampling process so that some old experiences can be re-learned, alleviating the problem of forgetfulness of the actor network.

[0211] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the above-mentioned task offloading. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiment of the task offloading device provided below can refer to the limitations of the task offloading method above, and will not be repeated here.

[0212] like Fig. 9 A task offloading device is shown, the device comprising:

[0213] An acquisition module 901 is used to acquire a directed acyclic graph task of each terminal device, and determine a current subtask based on the directed acyclic graph task, wherein the directed acyclic graph task includes a plurality of subtasks having a dependency relationship;

[0214] The strategy model module 902 is used to obtain the current state information of the edge computing system for each current subtask of the terminal device, input the current state information into the multi-objective reinforcement learning model, and generate a task offloading strategy corresponding to the current subtask;

[0215] The task execution module 903 is used for the terminal device and / or the server to perform a corresponding task offloading operation on the current subtask based on the task offloading rate in the task offloading strategy corresponding to the current subtask of each terminal device.

[0216] This implementation method determines the task offloading strategy of the current subtask of each terminal device based on a multi-objective reinforcement learning model, so that the tasks affected by the execution of each terminal device and each server in the edge computing system can be offloaded, and the connection between tasks can be fully considered. It can more comprehensively evaluate the interaction status of the terminal device and the edge server environment, the resource requirements of the task, and solve more complex task offloading problems, thereby improving the accuracy and efficiency of task offloading decisions.

[0217] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned task offloading methods is implemented.

[0218] Fig.10 It is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present application. Fig.10 The computer device shown includes: a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004. The processor 1001, the communication interface 1002 and the memory 1003 communicate with each other via the communication bus 1004. Fig.10The connection method between the processor 1001, the communication interface 1002, and the memory 1003 shown is merely exemplary. During implementation, the processor 1001, the communication interface 1002, and the memory 1003 may also be connected to each other in communication with each other using other connection methods besides the communication bus 1004.

[0219] The memory 1003 can be used to store computer programs, which may include instructions and data to implement the steps of any of the above task unloading methods. In an embodiment of the present application, the memory 1003 may be various types of storage media, such as random access memory (RAM), read only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, and registers. The memory 1003 may include a hard disk and / or a memory.

[0220] The processor 1001 may be a general-purpose processor, which may be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in a memory (e.g., memory 1003), and the general-purpose processor may use data stored in the memory (e.g., memory 1003) during the execution of the steps and / or operations. The general-purpose processor may be, for example, but not limited to, a central processing unit (CPU). In addition, the processor 1001 may also be a dedicated processor, which may be a processor specially designed to perform specific steps and / or operations, and the dedicated processor may be, for example, but not limited to, an ASIC and an FPGA, etc. In addition, the processor 1001 may also be a combination of multiple processors, such as a multi-core processor.

[0221] The communication interface 1002 may include an input / output (I / O) interface, a physical interface, and a logical interface for interconnecting devices within the network device, as well as an interface for interconnecting the network device with other devices (e.g., network devices). The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 1002 may be a module, a circuit, a transceiver, or any device capable of implementing communication.

[0222] In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 1001 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution. The software module can be located in a mature storage medium in the art such as a random access memory flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1003, and the processor 1001 reads the information in the memory 1003 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0223] Although the preferred embodiments of the present application have been disclosed for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the application as disclosed in the accompanying claims.

Claims

1. A task offloading method, applied to an edge computing system, wherein the edge computing system includes multiple terminal devices and multiple servers, characterized in that: The method comprises: Obtaining a directed acyclic graph task of each terminal device, and determining a current subtask based on the directed acyclic graph task, wherein the directed acyclic graph task includes a plurality of subtasks having a dependency relationship; For each current subtask of a terminal device, obtain the current state information of the edge computing system, input the current state information into a multi-objective reinforcement learning model, and generate a task offloading strategy corresponding to the current subtask, wherein the task offloading strategy includes a task offloading rate, and the terminal device for performing task offloading and the allocated computing resources are determined based on the task offloading rate; Based on the task offloading rate in the task offloading strategy corresponding to the current subtask of each terminal device, the terminal device and / or the server performs a corresponding task offloading operation on the current subtask; The current state information is input into the multi-objective reinforcement learning model to generate a task offloading strategy corresponding to the current subtask, including: For each directed acyclic graph task of a terminal device, a transmission network model of the directed acyclic graph task is constructed based on the network architecture between the edge computing system and the terminal device, a delay model of the directed acyclic graph task and a total energy consumption model of the terminal device are constructed based on the transmission network model, and an optimization model with the delay of the directed acyclic graph task and the total energy consumption of the terminal device as the target is constructed according to the delay model and the total energy consumption model; A Markov decision process is established based on the optimization model, and a state space, a mixed action space and a reward function of the Markov decision process are set, wherein the state space is a set of state information, and the state information includes the data size of the current subtask, the communication channel state, the available computing resources of the terminal device and the available computing resources of each server; the mixed action space includes a set of discrete actions and a set of continuous actions, and the discrete actions include: when the task offloading rate is a first value, all the current subtasks are offloaded to the terminal device, when the task offloading rate is a second value, all the current subtasks are offloaded to the target server, and when the task offloading rate is a value between the first value and the second value, a part of the current subtask is offloaded to the target server and another part of the current subtask is offloaded to the terminal device; the continuous action includes: the task offloading rate of the current subtask, and the computing resources allocated to the current subtask by the target server; the reward function includes a delay reward function and an energy consumption reward function; The state information at the current moment and the target weight vector at the current moment are input into the multi-objective reinforcement learning model, and the multi-objective reinforcement learning model outputs the selected discrete actions and continuous actions, and the task offloading strategy corresponding to the current subtask is determined by the discrete actions and continuous actions, wherein the target weight vector includes a delay weight vector and an energy consumption weight vector.

2. The task offloading method according to claim 1, characterized in that: A transmission network model of the directed acyclic graph task is constructed based on the network architecture between the edge computing system and the terminal device, and a delay model of the directed acyclic graph task and a total energy consumption model of the terminal device are constructed based on the transmission network model, including: For each subtask in the directed acyclic graph task, a first transmission rate from the terminal device to the server and a second transmission rate from the server to the terminal device are determined based on Shannon's theorem, a transmission time of the subtask is determined based on the data size of the subtask, the data size of the task result, the first transmission rate, and the second transmission rate, a transmission energy consumption of the subtask in the terminal device is determined based on the transmission time, and a transmission network model is constructed based on the transmission time and transmission energy consumption of each subtask; For each subtask in the directed acyclic graph task, a first task execution time of the subtask on the terminal device side is determined based on a task unloading rate of the subtask, a data size of the subtask, and resources allocated to the subtask by the terminal device; a second task execution time of the subtask on the server side is determined based on the task unloading rate of the subtask, the data size of the subtask, and resources allocated to the subtask by a server corresponding to the subtask; the maximum of the first task execution time and the second task execution time is used as the execution time of the subtask; the delay of the subtask is determined based on the transmission time and the execution time of the subtask, and then the execution delay of the directed acyclic graph task is determined; a task delay model is constructed based on the execution delay of the directed acyclic graph task; For each subtask in the directed acyclic graph task, the task execution energy consumption of the subtask on the terminal device is determined based on the first task execution time and the energy per unit time of the subtask, and the total energy consumption of the subtask is determined by the transmission energy consumption of the subtask on the terminal device and the task execution energy consumption of the subtask on the terminal device, and then the total energy consumption model of the terminal device is determined.

3. The task offloading method according to claim 1, characterized in that: The multi-objective reinforcement learning model includes an experience replay buffer, an actor network and a critic network, the actor network includes a discrete component and a continuous component, the critic network includes two Q networks and two target Q networks, the Q network and the target Q network have the same structure, and the multi-objective reinforcement learning model is trained by combining distributed execution with centralized training, including: The distributed execution method includes: Inputting the current state information and the current target weight vector into the discrete component and the continuous component in the actor network, wherein the discrete component outputs the selected discrete action, and the continuous component outputs the selected continuous action; The terminal device performs discrete actions and continuous actions to interact with the environment of the edge computing system; Obtaining state information at a next moment based on the interaction result, and determining a reward value obtained by the terminal device performing an action based on the reward function; The state information at the current moment, the discrete action, the continuous action, the target weight vector at the current moment, the state information at the next moment, and the reward value constitute an experience sample and store it in the experience playback buffer; The centralized training method includes: Using the experience samples in the experience replay buffer as training samples, and inputting the training samples into the critic network; The two Q networks respectively evaluate and output corresponding actual Q values ​​based on the actions output by the actor network, and use the minimum value among the output actual Q values ​​as the minimum actual Q value output by the critic network, and update the parameters of the actor network with the minimum actual Q value; The two target Q networks respectively evaluate and output corresponding target Q values ​​based on the action output by the actor network at the next moment, take the minimum value among the output target Q as the minimum target Q value, and update the parameters of the critic network based on the minimum actual Q value and the minimum target Q value.

4. The task offloading method according to claim 3, characterized in that: Updating the parameters of the actor network with the minimum actual Q value includes: Determine a discrete action loss function using a weight coefficient of the discrete action and a probability of outputting the discrete action and a minimum actual Q value output by the critic network; Determine a continuous action loss function using weight coefficients of continuous actions and probabilities of outputting continuous actions and a minimum actual Q value output by the critic network; Determining a first objective loss function of the actor network using the discrete action loss function and the continuous action loss function; Model training is performed based on the first objective loss function, gradient descent processing is performed on the first objective loss function, and network parameters of the actor network are updated according to a result of the gradient descent processing.

5. The task offloading method according to claim 3, characterized in that: Updating the parameters of the actor network with the minimum actual Q value includes: The discrete action loss function is determined by the weight coefficient of the discrete action and the probability of outputting the discrete action and the minimum actual Q value output by the critic network. for: ； in, Representing discrete actions The weight coefficient of It indicates the expectation of state s. Represents the state s and discrete actions The following actor network outputs the probability of discrete actions, represents the minimum practical Q value, and Represents parameters; The continuous action loss function is determined by the weight coefficient of the continuous action and the probability of outputting the continuous action and the minimum actual Q value output by the critic network. for: ； in, Indicates continuous action The weight coefficient of It indicates the expectation of state s. Represents the state s and discrete actions The actor network outputs continuous actions as follows The probability of Indicates the minimum practical Q value; With the discrete action loss function and the continuous action loss function Determine the first objective loss function of the actor network for: = + ； Model training is performed based on the first objective loss function, gradient descent processing is performed on the first objective loss function, and network parameters of the actor network are updated according to a result of the gradient descent processing.

6. The task offloading method according to claim 3, characterized in that: Updating the parameters of the critic network based on the minimum actual Q value and the minimum target Q value includes: Determine the joint entropy of the strategy based on the probability of the actor network outputting discrete actions and the probability of outputting continuous actions, and update the Bellman equation of the evaluation function of the critic network based on the joint entropy; The second objective loss function of the critic network is determined by the mean square error between the minimum actual Q value and the minimum target Q value, wherein the target Q value is obtained based on the following steps: the state information at the next moment and the target weight vector at the current moment are input into the actor network, the actor network outputs the action at the next moment, the state information at the next moment, the action at the next moment and the target weight vector at the current moment are input into the target Q network, the actual Q value at the next moment is obtained, and the target Q value is calculated by the Bellman equation; Model training is performed based on the second objective loss function, gradient descent processing is performed on the second objective loss function, and network parameters of the critic network are updated according to the result of the gradient descent processing.

7. The task offloading method according to claim 3, characterized in that: Updating the parameters of the critic network based on the minimum actual Q value and the minimum target Q value includes: The joint entropy of the strategy is determined by the probability of the actor network outputting discrete actions and the probability of outputting continuous actions. ; ； in, Representing discrete actions The weight coefficient of Represents the state s and discrete actions The actor network outputs discrete actions The probability of Indicates continuous action The weight coefficient of Represents the state s and discrete actions The actor network outputs continuous actions as follows The probability of Represents parameters; The Bellman equation for updating the critic network Q function based on the joint entropy is: ； in, Indicates that an action is performed in state s The instant rewards you receive, represents the discount factor, Indicates selecting the smallest Q value from two Q networks, Indicates the state at the next moment. Indicates the next continuous action. Represents a discrete action at the next moment; The second objective loss function of the critic network is determined by the mean square error between the minimum actual Q value and the minimum target Q value, wherein the target Q value is obtained based on the following steps: the state information of the next moment and the target weight vector of the current moment are input into the actor network, the actor network outputs the action of the next moment, the state information of the next moment, the action of the next moment and the target weight vector of the current moment are input into the target Q network, the actual Q value of the next moment is obtained, the target Q value is calculated by the Bellman equation, and the second objective loss function It is expressed as: ； in, represents the minimum target Q value, represents the minimum practical Q value, Indicates parameters; Model training is performed based on the second objective loss function, gradient descent processing is performed on the second objective loss function, and network parameters of the critic network are updated according to the result of the gradient descent processing.

8. The task offloading method according to claim 3, characterized in that: The historical experience in the experience playback buffer is used as a training sample, including: Sorting the experience samples in the experience replay buffer in a priority sequence based on the time difference error; Select λb experience samples as the first experience sample set based on the sorted experience samples; Based on the state error and the weight error between adjacent experience samples, the similarity between adjacent experience samples is calculated to determine the similarity of each experience sample in the first experience sample set; Determine the time correlation of each experience sample in the first experience sample set based on a time factor, wherein the time factor is set to , represents the exploration factor, T represents the current training time step, Indicates the time step number of the empirical sample sampling; Determine a new priority for each experience sample based on the similarity and time correlation of each experience sample; The first experience sample set is sorted in descending order based on the new priority, the first b experience samples are selected from the sorted first experience sample set for sampling, and the sampled samples are used as training samples.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the task offloading method according to any one of claims 1 to 8 are implemented.

Images