Depending task calculation unloading method meeting random preference of user

Abstract

The invention discloses a dependent task computing unloading method meeting random preference of a user, and belongs to the technical field of computing unloading of a mobile communication technology. In order to solve the problem that in the prior art, when a satellite depends on task calculation unloading, the requirement for random change of a user optimization target is often ignored, the application of a ground mobile terminal is divided into a workflow composed of a plurality of tasks, and the workflow is represented by a DAG; determining a scheduling sequence of the dependent tasks according to the DAG; constructing a random preference dependent task calculation unloading optimization model and function; converting a calculation unloading problem into an MDP decision process according to the created model, and solving an unloading decision by adopting multi-target reinforcement learning; and comprehensively considering the dynamic preference demand of the optimization target, and designing a composite reward function to obtain an optimal unloading decision. According to the method, the optimal unloading decision model meeting different preference requirements is provided, the solving efficiency and flexibility are improved, and the completion time and energy consumption of the dependent task are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile communication technology calculation offloading, and in particular relates to a method for offloading dependent task calculations that meets user random preferences. Background Art

[0002] The goal of 6G communication is to achieve global network coverage. The computing network refers to organizing and scheduling computing resources and services in a networked manner, so that users can easily obtain and use computing power distributed in different locations. At present, with the continuous increase in global terminal connections and data traffic, ground network computing alone can no longer effectively meet the needs of efficient and low-latency processing of massive data. Therefore, the integration of 6G and computing network research will be an integrated network with deep integration of ground network and satellite network. In order to provide ubiquitous, real-time and efficient Internet services, the combination of low-orbit satellite networks and ground networks, namely Satellite Terrestrial Networks (STN), has gradually become a trend. The satellite constellations currently planned and under construction include StarLink, OneWeb, TeleSat, SPUTNIX, Hongyan, etc. Satellite networks can provide backup network services for the ground, and can provide help when the ground network is unable to cope with heavy computing tasks. Specifically, 6G networks will further integrate satellite communications, big data, artificial intelligence and other technologies on the basis of 5G technology to provide a wider coverage, greater communication capacity and lower transmission latency. They will have higher flexibility and scalability, and will be able to randomly manage and configure computing resources according to user needs, thereby reducing computing costs and risks, allowing users to easily obtain and use computing power technologies distributed in different locations.

[0003] With the rapid development of the Internet of Things and wireless communication technologies, emerging computing applications are becoming more complex. These applications usually involve multiple interdependent computing tasks that must be executed in a specific order to ensure the smooth completion of the entire business process. For example, the design of mobile applications usually includes multiple tasks, such as data collection, processing, and transmission, which have complex dependencies. This complex dependency structure makes it crucial to make online offloading decisions under the limited computing resources of edge servers. Therefore, how to formulate efficient computing offloading strategies based on limited network resources to meet the needs of emerging computing applications and ensure the efficient completion of the entire task process has become the current research focus. Most of the existing research on computing power scheduling for satellite edge computing assumes a static computing offloading scenario, in which the computing tasks of each user device need to be completed under strict latency or energy consumption requirements. Static offloading scenarios are convenient for theoretical analysis, but can only reflect short-term performance. Due to the high-speed movement of satellites and the frequent switching of ground user terminals, static scenarios lack research on random changes in satellite computing offloading. At the same time, when studying satellite-dependent task computing offloading, existing methods often ignore the problem of random changes in user optimization objectives. Most studies only minimize latency or energy consumption, and some comprehensive trade-offs do not consider the user's random preference requirements for task computing latency and energy consumption, and only rely on converting multiple objectives into a single objective to establish a model. Since users have different preferences for different objectives at different times, it is difficult to determine the appropriate weights, so the results of these methods are difficult to meet user needs. Summary of the invention

[0004] In view of the fact that existing methods often ignore the problem of random changes in user optimization objectives when studying satellite dependent task computational offloading, the present invention provides a dependent task computational offloading method that meets the user's random preferences, studies the computational offloading strategy under the user's random preference requirements for dependent tasks, and considers the offloading optimization problem of multiple dependent subtasks that change with the high-speed operation of the satellite.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Step 1: Model the satellite network and user computing model, divide the application of the ground mobile terminal into a workflow consisting of multiple tasks, use a DAG graph to represent it, and determine the scheduling order of dependent tasks based on the DAG graph;

[0007] The set of low-orbit satellite servers is represented by M. The computing task requested by the ground terminal user consists of multiple subtasks with dependencies. The dependencies between subtasks are represented by DAG graph G = (v, e): v represents the vertex set, each vertex represents a subtask, and e represents the dependency between subtasks, that is, a subtask can only start computing after its predecessor subtask is completed;

[0008] In order to better describe the dependency and calculation process of subtasks, the direct predecessor subtask set pred(i) and the direct successor subtask set succ(i) of the subtask are defined;

[0009] The task requested by the user is composed of multiple subtasks with dependencies. The resulting series of subtasks is represented by K = {K i |1≤i≤N}, Ki is the i-th subtask of the device, defined as K i ={D i ,S i ,T i max}; The elements in it represent the data size of the i-th subtask, the number of CPU cycles required to calculate 1 bit, and the maximum delay that the subtask can accept;

[0010] Each subtask adopts a binary offloading method, and the action space of each subtask in time slot t is in b i ∈{0,1,...,M}, represents the subtask K generated by the ground terminal user i Compute locally, This means that the subtask is offloaded to the low-orbit satellite server for calculation; b i =M means that the subtask is offloaded to the Mth satellite edge node for calculation through the satellite-to-ground link;

[0011] The DAG graph of the user's dependent task workflow is topologically sorted according to the depth-first traversal to determine the order and dependency relationship between each subtask, and finally obtain the task scheduling execution sequence;

[0012] Step 2: Construct a dynamic preference-dependent task communication model, computational offloading model, and satellite-to-ground communication time model for satellite networks;

[0013] For each subtask in the set of dependent task sequences, define T i loc , and They represent the completion time of local computing, task sending, and edge node computing, respectively, as shown below:

[0014]

[0015]

[0016] Among them, f i L For local subtask K iAllocated computing power, Indicates that low-orbit satellite j is dependent on subtask K i Allocated computing power, d i,j Represents the dependent subtask K i The distance between the ground terminal device and the low-orbit satellite j, c represents the speed of light, V i,j Represents the dependent subtask K generated by the terminal device i The uplink rate of computation offloading to LEO satellite j through the satellite-to-ground link;

[0017] Local computing is divided into the following two cases based on the computing latency and energy consumption of the predecessor task:

[0018] When its predecessor subtask is executed locally, the dependent subtask K i The calculation delay FT i l for: Dependent subtask K i Computational energy consumption for:

[0019] Among them, K i' Represents subtask K i The predecessor task of Represents the predecessor subtask K i' The local computation time, P i l Subtask K generated by the ground terminal user i Local operating power, Indicates the dependent subtask K i' The local computing latency, The ground terminal user generates subtask K i' The local operating power, pred(K i ) represents subtask K i The predecessor subtask of

[0020] When its predecessor task is to offload computation execution, it depends on subtask K i The calculation delay FT i l for: Dependent subtask K i Computational energy consumption for:

[0021] in, Represents the predecessor subtask K i' The time it takes to offload data transmission to the edge server, Represents the predecessor subtask K i' The time consumed by edge server j to calculate;

[0022] The computational offloading model is divided into the following two cases according to the computational latency and energy consumption of the predecessor task:

[0023] When its predecessor task is executed locally, it depends on subtask K i The calculation delay FT i m for: Dependent subtask K i Computational energy consumption for:

[0024] When its predecessor task is to offload computation execution, it depends on subtask K i The computational delay for: Dependent subtask K i Computational energy consumption for:

[0025]

[0026] Low-orbit satellite coverage model: Ground terminal equipment can only establish a connection and communicate with the low-orbit satellite within its coverage area; θ i,j is the elevation angle between ground terminal equipment i and low-orbit satellite j. The elevation angle formula is:

[0027]

[0028] Among them, R represents the radius of the earth, h represents the dependent subtask K i The distance between the edge nodes of the low-orbit satellite orbit, γ i,j is the LEO satellite j relative to the dependent subtask K i The geocentric angle of the coverage area, expressed as:

[0029]

[0030] set up is the longest communication time after the connection is established between the ground terminal device i and the low-orbit satellite j, expressed as:

[0031]

[0032] Among them, V S Indicates the speed of the low-orbit satellite, L i,j is the arc length corresponding to the connection between ground terminal device i and low-orbit satellite j. The arc length is expressed as: L i,j =2(R+h)·γ i,j .

[0033] Step 3: According to the constraints observed in the process of minimizing system delay and energy consumption, a dynamic preference dependent task calculation offloading optimization function is constructed;

[0034] Furthermore, the constraints in step 3 are:

[0035] Constraint 1: Subtask K i The completion time of the task must not exceed the maximum tolerance time of the task under the offloading decision;

[0036] Constraint 2: Subtask K i The completion time of must not exceed the maximum coverage time of low-orbit satellite j under the unloading decision;

[0037] Constraint 3: Low-orbit satellite j is a subtask K of ground equipment i The allocated computing resources do not exceed its own maximum computing power;

[0038] Constraint 4: Delay on ground terminal equipment subtask K i the importance of

[0039] Constraint 5: represents the ground terminal subtask K i The offloading decision indicates whether to offload the calculation and to which low-orbit satellite server the calculation is offloaded;

[0040] Constraint 6: This subtask can only be executed after its predecessor task is completed;

[0041] Then the dynamic preference dependent task calculation offloading optimization function is expressed as:

[0042]

[0043] Among them, w i Represents subtask K i The preference value for delay, Indicates the dependent subtask K i The time consumed by local calculation, T i max Indicates the maximum delay that the subtask can accept. Indicates the maximum computing capacity of the satellite node, RT i m Indicates the dependent subtask K i Offload the time spent on calculations, Indicates the dependent subtask K i' The time spent on offloading calculations, i' represents the predecessor subtask K i' The subscript of .

[0044] Step 4: Use deep reinforcement learning to solve the computational offloading optimization function of the low-orbit satellite mobile edge computing network. The solution is as follows: describe the computational offloading problem as a multi-objective Markov decision process (S, A, P, r, w), and transform it into a problem of solving the optimal computational offloading control strategy;

[0045] Furthermore, the computation offloading problem is described as a Markov decision process (S, A, P, r, w), which is transformed into a problem of solving the optimal computation offloading control strategy. The deep reinforcement learning method is used to solve it, determine the entity of the intelligent agent and set the Markov state space, action space and reward function, where S represents the state space, A represents the action space, P represents the state transition probability, r is the reward, and w represents the preference space that stores different preference values. In state S, the action A is selected according to the reinforcement learning algorithm to obtain an optimal computation offloading control strategy π* to maximize the long-term cumulative reward obtained during the device movement process.

[0046] The state space of a certain time slot is defined as Where DAG(t) represents the dependency relationship between user subtasks at time slot t, and D i (t) represents the data size of the ith subtask of the terminal device at time slot t, f i L (t) represents the local computing power allocated by the ground terminal device to its generated subtask Ki, represents the computing power of low-orbit satellite j at time slot t, w i (t) represents the subtask K in time slot t i The preference value for delay, K i Indicates the i-th subtask of the device;

[0047] The action space of a certain time slot is defined as in b i ∈{0,1,...,M}, represents the subtask K generated by the ground terminal user i Compute locally, This means that the subtask is offloaded to the low-orbit satellite server for calculation; b i =M means that the subtask is offloaded to the Mth satellite edge node for calculation through the satellite-to-ground link.

[0048] The compound reward function is set up as follows:

[0049]

[0050] The agent uses the ε-greedy method to select the execution action in combination with the current network state S, selects the action corresponding to the maximum value of the multiplication result of the current preference and the Q vector in the optional action space of each task with a probability of ε, and selects a random action with a probability of 1-ε:

[0051]

[0052] The objective function is set up as follows:

[0053]

[0054] where γ is the discount factor, It indicates that the action corresponding to the maximum value of the multiplication result of the preference and Q vector is selected in the main network, θ is the parameter of the main network, and θ′ is the parameter of the target network.

[0055] The loss function is:

[0056] L(θ t )=E s,a,w [(y t -Q(s t ,a t ,w;θ t )) 2 ]

[0057] In the above formula, Q represents the Q value obtained by the Q network, s′ is the next state output by the Q network, and the Q network is updated using the loss function value, and the Q network parameters are synchronized to the target Q network every 300 iterations.

[0058] Step 5: Initialize the parameters of the main network and target network in deep reinforcement learning, build an experience pool of size d, set the number of iterations, initialize the state, and use the random method to generate preference values, so as to assign the current user preference value to the task optimization target, laying the foundation for the training process;

[0059] Step 6: Use random sampling to select samples from the experience pool to train the deep reinforcement learning algorithm;

[0060] Step 7: Obtain the system status and preference of the current time slot, input the system status into the trained deep reinforcement learning algorithm, and use the trained deep reinforcement learning algorithm to obtain the task offloading decision for each time slot.

[0061] Furthermore, the system state, system action, immediate reward, preference of the current time slot and the system state of the next time slot obtained from the training are stored in the experience pool, and m samples are randomly sampled from the sample pool in the experience buffer pool for training. The loss function is minimized and the main network parameters are updated according to the gradient descent method. During the training process, the input of the network is the current state s and preference, and the output is the Q vector. The training aims to maximize the reward and obtain the optimal unloading decision. After the iteration, the long-term cumulative reward reaches the optimal, and the optimal unloading decision can be obtained.

[0062] Compared with the prior art, the present invention has the following advantages:

[0063] 1. Study the dynamic differences in user task optimization target preferences. Traditional edge computing research often ignores the changes in user task preferences over time. However, in practical applications, users' preference requirements for tasks are diverse and constantly changing, resulting in dynamic adjustments to optimization targets. To solve this problem, the present invention proposes a method for offloading dependent task calculations that meets the user's random preferences based on the characteristics of dependent tasks of ground terminal devices, aiming to more effectively meet the ever-changing user optimization needs and improve the overall performance of computation offloading.

[0064] 2. Pay attention to the time limit of satellite-to-ground communication caused by the high-speed operation of low-orbit satellites. The present invention introduces low-orbit satellites as edge nodes, focusing on the communication delay limit brought by their high-speed operation, and making full use of their wider coverage advantages. The introduction of low-orbit satellites not only provides more flexible edge computing resources for computing offloading, but also effectively shortens the delay of satellite-to-ground communication, laying the foundation for improving the overall system performance.

[0065] 3. Solve the computational offloading decision problem of user tasks in complex networks based on deep reinforcement learning. In order to overcome the shortcomings of traditional methods that run a complete optimization algorithm for each set of optimization problems, this paper introduces deep reinforcement learning to effectively cope with dynamic network environments; by inputting user preferences and network status, deep reinforcement learning can generate the optimal computational offloading strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the process of the present invention;

[0067] Figure 2 It is the reward convergence diagram of the algorithm of the present invention;

[0068] Figure 3 It is the loss function convergence diagram of the algorithm of the present invention;

[0069] Figure 4 This is a comparison chart of various indicators between the present invention and other methods. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] A method for offloading dependent task calculations that meets user random preferences. The flowchart is shown in the attached figure. Figure 1 As shown, the specific implementation steps are as follows:

[0072] Step 1: Build satellite network systems and models

[0073] In order to realize the multi-objective reinforcement learning computation offloading function for random preference tasks in low-orbit satellite constellation networks, this step builds a satellite-assisted MEC system, in which each low-orbit satellite carries a MEC server to offload tasks to users within the coverage area. The computation offloading problem is modeled as a multi-objective optimization problem, and the low-orbit satellite coverage time constraint is added to minimize the multiple objectives of satellite network system latency and energy consumption at the same time.

[0074] This step models the satellite network and user model. The specific method is as follows: The present invention constructs a satellite network computing offloading system, using orthogonal frequency division multiple access (OFDMA) wireless communication technology. The set of all satellite edge servers is represented by M. The task requested by the user is composed of multiple subtasks with dependencies. The dependencies between subtasks are represented by a directed acyclic graph G = (v, e): v represents a vertex set, each vertex represents a subtask, and e represents the dependency between subtasks, that is, a subtask can only start computing after its predecessor subtask is completed. In order to better describe the dependencies and computing process of subtasks, the direct predecessor subtask set pred(i) and the direct successor subtask set succ(i) of the subtask are defined. The resulting series of subtask sets can be expressed as K = {K i |1≤i≤N}, where N represents the number of subtasks of the user terminal device, K i is the i-th subtask of the device, defined as K i ={D i ,S i ,T i max}, the elements in it represent the data size of the i-th subtask, the number of CPU cycles required to calculate 1 bit, and the maximum delay that the subtask can accept.

[0075] The end user makes intelligent migration decisions based on the generated task information and the availability of computing resources and communication resources in the edge network. Each subtask adopts a binary offloading method. The offloading strategy of each subtask in time slot t is in b i ∈{0,1,...,M}, represents the subtask K generated by the ground terminal user i Compute locally, This means that the subtask is offloaded to the low-orbit satellite server for calculation; b i =M means that the subtask is offloaded to the Mth satellite edge node for calculation through the satellite-to-ground link.

[0076] The communication model, computation offloading model and low-orbit satellite coverage model of low-orbit satellite computation offloading are described below respectively.

[0077] Low-orbit satellite communication model: Assume that each ground terminal device can only transmit task data offload with one low-orbit satellite, multiple ground terminal devices share the same spectrum resources, and different terminal devices offload task calculations to different edge nodes with different uplink and downlink rates. Then the dependent subtasks K generated by the terminal devices i The uplink rate of computation offloading to edge node j via the satellite-to-ground link is as follows:

[0078]

[0079] Where B is the available spectrum bandwidth allocated by LEO satellite edge node j to user terminals, g i,j Represents subtask K i The channel power gain to the LEO satellite edge node j, P i up For subtask K i The transmission power in the uplink, σ 2 is the Gaussian white noise power. In addition, considering that the size of the calculation result is much smaller than the size of the input calculation data, and the downlink transmission rate of the low-orbit satellite is much greater than the upload rate of the ground user. Therefore, the present invention ignores the downlink transmission delay caused by the low-orbit satellite transmitting the calculation result to the ground user.

[0080] Computation offloading model: For each subtask in the set of dependent task sequences, define and They represent the completion time of local computing, task sending, and edge node computing, respectively, as shown below:

[0081]

[0082] Among them, f iL For local subtask K i Allocated computing power, Indicates that low-orbit satellite j is dependent on subtask K i Allocated computing power, d i,j Represents the dependent subtask K i The distance between the ground terminal device and the low-orbit satellite j, c represents the speed of light, V i,j Represents the dependent subtask K generated by the terminal device i The uplink rate of computation offloading to LEO satellite j through the satellite-to-ground link;

[0083] The local computing model is divided into the following two cases according to the computing latency and energy consumption of the predecessor task:

[0084] If its predecessor task is executed locally, it depends on subtask K i The calculation delay is: Dependent subtask K i The computational energy consumption is:

[0085] When its predecessor subtask is executed locally, the dependent subtask K i The calculation delay FT i l for: Dependent subtask K i Computational energy consumption for:

[0086] Among them, K i' Represents subtask K i The predecessor task of Represents the predecessor subtask K i' The local computation time, P i l Subtask K generated by the ground terminal user i Local operating power, Represents the dependent subtask K i' The local computing latency, The ground terminal user generates subtask K i' The local operating power, pred(K i ) represents subtask K i The predecessor subtask of

[0087] When its predecessor subtask is to offload computation execution, the dependent subtask K i The calculation delay FT i l for: Dependent subtask K i Computational energy consumption for:

[0088] in, Represents the predecessor subtask K i' The time it takes to offload data transmission to the edge server, Represents the predecessor subtask K i' The time consumed by edge server j to calculate;

[0089] The computation offloading model is divided into the following two cases according to the computation delay and energy consumption of the predecessor subtask:

[0090] When its predecessor subtask is executed locally, the dependent subtask K i The calculation delay FT i m for: Dependent subtask K i Computational energy consumption for:

[0091] When its predecessor subtask is to offload computation execution, the dependent subtask K i The computational delay for: Dependent subtask K i Computational energy consumption for:

[0092]

[0093] Among them, P i up Represents the dependent subtask K i Uplink transmission power;

[0094] Low-orbit satellite coverage model: Ground terminal equipment can only establish a connection and communicate with the low-orbit satellite within its coverage area; θ i,j is the elevation angle between ground terminal equipment i and low-orbit satellite j. The elevation angle formula is:

[0095]

[0096] Among them, R represents the radius of the earth, h represents the dependent subtask K i The distance between the edge nodes of the low-orbit satellite orbit, γ i,j is the LEO satellite j relative to the dependent subtask K i The geocentric angle of the coverage area, expressed as:

[0097]

[0098] set up is the dependent subtask K iThe longest communication time after establishing a connection with low-orbit satellite j is expressed as:

[0099]

[0100] Where V S Indicates the speed of the low-orbit satellite, L i,j is the dependent subtask K i The arc length corresponding to establishing a connection with a low-orbit satellite j is expressed as: L i,j =2(R+h)·γ i,j .

[0101] Step 2: Under the constraints of full consideration of user random preference requirements, satellite limited computing power, and task dependencies, minimize the task execution delay and total energy consumption of all users, and construct a random preference dependent task calculation offloading optimization function.

[0102] In addition, in the process of minimizing system latency and energy consumption, we also need to follow the following constraints:

[0103] Constraint 1: Subtask K i The completion time of the task must not exceed the maximum tolerance time of the task under the offloading decision;

[0104] Constraint 2: Subtask K i The completion time of must not exceed the maximum coverage time of low-orbit satellite j under the unloading decision;

[0105] Constraint 3: Low-orbit satellite j is a subtask K of ground equipment i The allocated computing resources do not exceed its own maximum computing power;

[0106] Constraint 4: Delay on ground terminal equipment subtask K i the importance of

[0107] Constraint 5: represents the ground terminal subtask K i The offloading decision indicates whether to offload the calculation and to which low-orbit satellite server the calculation is offloaded;

[0108] Constraint 6: Indicates that this subtask can only be executed after its predecessor task is completed.

[0109] The above constrained optimization problem can be expressed as:

[0110]

[0111] Among them, w i Represents subtask K i The preference value for delay, Represents the dependent subtask K i The time consumed by local calculation, Ti max Indicates the maximum delay that the subtask can accept. Indicates the maximum computing capacity of the satellite node, RT i m Indicates the dependent subtask K i Offload the time spent on calculations, Indicates the dependent subtask K i' The time spent on offloading calculations, i' represents the predecessor subtask K i' The subscript of .

[0112] Step 3: Describe the computational offloading problem as a Markov decision process and transform it into using deep reinforcement learning to solve the optimal offloading strategy.

[0113] The specific implementation method of this step is as follows: first, describe the computation offloading problem as a Markov decision process (S, A, P, r, w), transform it into a problem of solving the optimal computation offloading control strategy, and use the deep reinforcement learning method to solve it, determine the entity of the intelligent agent and set the Markov state space, action space and reward function, where S represents the state space, A represents the action space, P represents the state transition probability, r is the reward, and w represents the preference space that stores different preference values. Under state S, the action A is selected according to the reinforcement learning algorithm to obtain an optimal computation offloading control strategy π* to maximize the long-term cumulative reward obtained during the device movement process;

[0114] The state space of a certain time slot is defined as Where DAG(t) represents the dependency relationship between user subtasks at time slot t, and D i (t) represents the data size of the ith subtask of the terminal device at time slot t, f i L (t) represents the local computing power allocated by the ground terminal device to its generated subtask Ki, represents the computing power of LEO satellite j at time slot t, w i (t) represents the subtask K in time slot t i The preference value for delay, K i Indicates the i-th subtask of the device;

[0115] The action space of a certain time slot is defined as in b i ∈{0,1,...,M}, represents the subtask K generated by the ground terminal user i Compute locally, This means that the subtask is offloaded to the low-orbit satellite server for calculation; b i= M means that the subtask is offloaded to the Mth satellite edge node through the satellite-to-ground link for calculation

[0116] Step 4: Initialize the neural network parameters, experience pool size, generate preference values ​​using random methods, and assign current user preferences (weights) to the latency optimization target. The neural network in the deep reinforcement learning includes a main network and a target network, θ is the main network parameter, and θ′ is the target network parameter.

[0117] Step 5: The agent in deep reinforcement learning starts to interact with the MEC environment. On the one hand, the agent obtains the current state from the environment. On the other hand, the environment returns the current reward value and the next state through the action selected by the agent, and updates the experience carrying the preference to the experience pool. The composite reward function is set as follows:

[0118]

[0119] The agent uses the ε-greedy method to select the execution action in combination with the current network state s, selects the action corresponding to the maximum value of the multiplication result of the current preference and the Q vector in the optional action space of each task with a probability of ε, and selects a random action with a probability of 1-ε:

[0120]

[0121] Step 6: Determine whether the sample buffer pool overflows

[0122] If there is an overflow, the five-tuple <s, a, r, w, s'> is stored sequentially in the experience pool; otherwise, the five-tuple <s, a, r, w, s'> is randomly stored in the experience pool with replacement samples.

[0123] Step 7: Randomly sample m samples from the sample pool and conduct training to maximize the reward and obtain the optimal unloading decision. During the training process, the input of the network is the current state and preference, and the output is the Q vector.

[0124] Step 8: Calculate the objective function and loss function

[0125] The objective function is:

[0126]

[0127] Where γ is the discount factor, It indicates that the action corresponding to the maximum value of the multiplication result of the preference and Q vector is selected in the main network, θ is the parameter of the main network, and θ′ is the parameter of the target network.

[0128] The loss function is:

[0129] L(θ t )=E s,a,w[(y t -Q(s t ,a t ,w;θ t )) 2 ]

[0130] In the above formula, Q represents the Q value obtained by the Q network, γ represents the reward discount factor, s′ is the next state output by the Q network, and the Q network is updated using the loss function value, and the Q network parameters are synchronized to the target Q network every 300 iterations;

[0131] Step 9: Update the parameters θ of the main network using the gradient descent method.

[0132] The gradient descent method (LD) can be expressed as ▽ θ L is to derive along all the parameters of the main network to obtain the direction of minimizing the objective function.

[0133] Step 10: When the preset period is reached, parameter replication is performed, i.e., θ′←θ

[0134] The preset period is 300. After each 300th iteration, the values ​​of all parameters of the main network are copied to the corresponding target network as all parameter values ​​of the target network, so as to achieve the purpose of regular update and replacement.

[0135] Step 11: If the optimal strategy is obtained, stop training. The optimal strategy is represented by π*, which refers to the action a selected when the maximum cumulative reward is obtained, and can be expressed as the computational offloading and resource allocation strategy that obtains the maximum cumulative reward.

[0136] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

[0137] The effect of the present invention is verified by simulation experiments below:

[0138] The simulation experiment was conducted in PyCharm, with Python 3.8 and Pytorch 1.8.2 as the software environment. It was set as a ground terminal device with 8 dependent subtask sequence sets. Each user can only offload data transmission with one satellite edge server at a time. The task data volume is D∈[50,100]MB, and the number of CPU operation clock cycles required for the ground user terminal and edge server to complete the calculation of 1bit task data is 100cycle / bit. The terminal device can choose to process subtasks locally or offload these subtasks to different satellite edge nodes for parallel calculation. There are 3 low-orbit satellites carrying MEC servers at an altitude of 1000KM in the orbit above the ground terminal device. The computing power of each satellite carrying server is uniformly distributed between [5,15]GHZ, and the flight speed is 3km / s. The size of the dependent task data generated by the user is D∈[50,100]MB, and the number of CPU cycles required to complete the task is set to 1000cycles / bit. The elevation angle range between the user terminal and the low-orbit satellite is set to [10,30], and the radius of the earth is 6371KM. According to the algorithm performance under different neural network parameters, BATCHSIZE=64 and MEMORY CAPACITY=5000 are selected.

[0139] Figure 2 The reward convergence of the present invention after multiple iterations is demonstrated. In the learning and training process of the agent, the agent is in the trial and error stage at the beginning, so the reward shows a fluctuating state. As the agent interacts with the environment multiple times and continuously adjusts the neural network parameters, the agent gradually begins to understand the characteristics of the environment and the reward signal. As the learning deepens, the agent gradually optimizes its behavior strategy to achieve higher reward benefits. Finally, the agent successfully implements the optimal collaboration strategy and achieves the convergence of the training process by adjusting the neural network parameters and the learned knowledge.

[0140] Figure 3 The convergence of the loss function of the present invention after multiple iterations is demonstrated. In the initial training stage, due to the setting of the initial parameters of the neural network, the agent exhibited an unstable learning strategy when facing an unknown environment, resulting in fluctuations in the loss function value. After multiple iterations, the loss function value gradually stabilized. This shows that the agent gradually mastered the environmental characteristics and reward signals during the learning process, optimized the decision-making strategy, and enabled it to perform actions more effectively to obtain higher rewards. By adjusting the strategy to minimize the loss function, the agent learned a more robust and optimized decision-making method from the initial unstable strategy.

[0141] Figure 4A comparison chart of various indicators of the present invention and other methods under the same environment is shown. It can be seen from the figure that the local computing method chooses to complete the calculation of all dependent tasks locally, but due to its limited computing resources, it leads to high latency and energy consumption. The genetic algorithm searches for the global optimal solution in the entire search space, which is suitable for static and smaller-scale task computing scenarios. It is difficult to quickly adapt to find the optimal solution in dynamically changing computing offloading scenarios, resulting in higher average latency and energy consumption. The reinforcement learning method (RL) proposed in the present invention can better capture the complex relationship between tasks through the dynamic adjustment strategy of the neural network, can flexibly adapt to different computing environments and needs, show stronger learning ability, and show low latency and energy consumption.

[0142] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for offloading dependent task calculations that meets user random preferences, characterized in that: The method comprises the following steps: Step 1: Model the satellite network and user computing model, divide the application of the ground mobile terminal into a workflow consisting of multiple tasks, use a DAG graph to represent it, and determine the scheduling order of dependent tasks based on the DAG graph; Step 2: Construct a dynamic preference-dependent task communication model, computational offloading model, and satellite-to-ground communication time model for satellite networks; Step 3: According to the constraints observed in the process of minimizing system delay and energy consumption, a dynamic preference dependent task calculation offloading optimization function is constructed; Step 4: Use deep reinforcement learning to solve the dynamic preference dependent task calculation offloading optimization function; Step 5: Initialize the parameters of the main network and target network in deep reinforcement learning, build an experience pool of size d, set the number of iterations, initialize the state, and use random method to generate preference values; Step 6: Use random sampling to select samples from the experience pool to train the deep reinforcement learning algorithm; Step 7: Obtain the system status and preference of the current time slot, input the system status into the trained deep reinforcement learning algorithm, and use the trained deep reinforcement learning algorithm to obtain the task offloading decision for each time slot.

2. A method for offloading dependent task calculations that satisfies user random preferences according to claim 1, characterized in that: The specific operations of step 1 are: The set of low-orbit satellite servers is represented by M. The computing task requested by the ground terminal user consists of multiple subtasks with dependencies. The dependencies between subtasks are represented by DAG graph G = (v, e): v represents the vertex set, each vertex represents a subtask, and e represents the dependency between subtasks, that is, a subtask can only start computing after its predecessor subtask is completed; Define the direct predecessor subtask set pred(i) and the direct successor subtask set succ(i) of the subtask; The task requested by the user is composed of multiple subtasks with dependencies. The resulting series of subtasks is represented by K = {K i |1≤i≤N},K i is the i-th subtask of the device, defined as K i ={D i ,S i ,T i max }; The elements in it represent the data size of the i-th subtask, the number of CPU cycles required to calculate 1 bit, and the maximum delay that the subtask can accept; Each subtask adopts a binary offloading method, and the action space of each subtask in time slot t is in b i ∈{0,1,...,M}, represents the subtask K generated by the ground terminal user i Compute locally, This means that the subtask is offloaded to the low-orbit satellite server for calculation; b i =M means that the subtask is offloaded to the Mth satellite edge node for calculation through the satellite-to-ground link; The DAG graph of the user-dependent task workflow is topologically sorted according to the depth-first traversal to determine the sequence and dependency relationship between each subtask, and finally obtain the task scheduling execution sequence.

3. A method for offloading dependent task calculations that satisfies user random preferences according to claim 2, characterized in that: In the step 2, in constructing the dynamic preference dependent task communication model, computation offloading model and satellite-to-ground communication time model for satellite networks, for each subtask in the dependent task sequence set, define and They represent the completion time of local computing, task sending, and edge node computing, respectively, as shown below: Among them, f i L Indicates that the local is subtask K i Allocated computing power, Indicates that low-orbit satellite j is dependent on subtask K i Allocated computing power, d i,j Indicates the dependent subtask K i The distance between the ground terminal device and the low-orbit satellite j, c represents the speed of light, V i,j Represents the dependent subtask K generated by the terminal device i The uplink rate of computation offloading to LEO satellite j through the satellite-to-ground link; Local computing is divided into the following two cases based on the computing latency and energy consumption of the predecessor subtask: When its predecessor subtask is executed locally, the dependent subtask K i The calculation delay FT i l for: Dependent subtask K i Computational energy consumption for: Among them, K i' Represents subtask K i The predecessor task of Represents the predecessor subtask K i' The local computation time, P i l Subtask K generated by the ground terminal user i Local operating power, Indicates the dependent subtask K i' The local computing latency, The ground terminal user generates subtask K i' The local operating power, pred(K i ) represents subtask K i The predecessor subtask of When its predecessor subtask is to offload computation execution, the dependent subtask K i The calculation delay FT i l for: Dependent subtask K i Computational energy consumption for: in, Represents the predecessor subtask K i' The time it takes to offload data transmission to edge server j, Represents the predecessor subtask K i' The time consumed by edge server j to calculate; The offloading calculation model is divided into the following two cases according to the calculation delay and energy consumption of the predecessor subtask: When its predecessor subtask is executed locally, the dependent subtask K i The calculation delay FT i m for: Dependent subtask K i Computational energy consumption for: When its predecessor subtask is to offload computation execution, the dependent subtask K i The calculation delay RT k m for: Dependent subtask K i The computational energy consumption FT k m for: Among them, P i up Indicates the dependent subtask K i Uplink transmission power; Low-orbit satellite coverage model: Ground terminal equipment can only establish a connection and communicate with the low-orbit satellite within its coverage area; θ i,j is the dependent subtask K i The elevation angle between the low-orbit satellite j and the elevation angle formula is: Among them, R represents the radius of the earth, h represents the dependent subtask K i The distance between the edge nodes of the low-orbit satellite orbit, γ i,j is the LEO satellite j relative to the dependent subtask K i The geocentric angle of the coverage area, expressed as: set up is the dependent subtask K i The longest communication time after establishing a connection with low-orbit satellite j is expressed as: Where V S Indicates the speed of the low-orbit satellite, L i,j is the dependent subtask K i The arc length corresponding to establishing a connection with a low-orbit satellite j is expressed as: L i,j =2(R+h)·γ i,j .

4. A method for offloading dependent task calculations that satisfies user random preferences according to claim 3, characterized in that: The constraints in step 3 are: Constraint 1: Subtask K i The completion time of the task must not exceed the maximum tolerance time of the task under the offloading decision; Constraint 2: Subtask K i The completion time of must not exceed the maximum coverage time of low-orbit satellite j under the unloading decision; Constraint 3: Low-orbit satellite j is a subtask K of ground equipment i The allocated computing resources do not exceed its own maximum computing power; Constraint 4: Delay on ground terminal equipment subtask K i the importance of Constraint 5: represents the ground terminal subtask K i The offloading decision indicates whether to offload the calculation and to which low-orbit satellite server the calculation is offloaded; Constraint 6: This subtask can only be executed after its predecessor task is completed; Then the dynamic preference dependent task calculation offloading optimization function is expressed as: Among them, w i Represents subtask K i The preference value for delay, Indicates the dependent subtask K i The time consumed by local calculation, T i max Indicates the maximum delay that the subtask can accept. Indicates the maximum computing capacity of the satellite node, RT i m Indicates the dependent subtask K i Offload the time spent on calculations, Indicates the dependent subtask K i' The time spent on offloading calculations, i' represents the predecessor subtask K i' The subscript of .

5. A method for offloading dependent task calculations that satisfies user random preferences according to claim 4, characterized in that: The specific operation of step 4 is: describing the computation offloading problem as a Markov decision process (S, A, P, r, w), transforming it into a problem of solving the optimal computation offloading control strategy, solving it by using a deep reinforcement learning method, determining the entity of the intelligent agent and setting the Markov state space, action space and reward function, where S represents the state space, A represents the action space, P represents the state transition probability, r is the reward, and w represents the preference space for storing different preference values, selecting the action A to be executed according to the reinforcement learning algorithm in state S, and obtaining an optimal computation offloading control strategy π* to maximize the long-term cumulative reward obtained during the device movement process; The state space of a certain time slot is defined as Where DAG(t) represents the dependency relationship between user subtasks at time slot t, and D i (t) represents the data size of the ith subtask of the terminal device at time slot t, f i L (t) represents the subtask K generated by the ground terminal device i Allocated local computing power, represents the computing power of low-orbit satellite j at time slot t, w i (t) represents the subtask K in time slot t i The preference value for delay, K i Indicates the i-th subtask of the device; The action space of a certain time slot is defined as in b i ∈{0,1,...,M}, represents the subtask K generated by the ground terminal user i Compute locally, This means that the subtask is offloaded to the low-orbit satellite server for calculation; b i =M means that the subtask is offloaded to the Mth satellite edge node for calculation through the satellite-to-ground link; The compound reward function is set up as follows: The agent uses the ε-greedy method to select the execution action in combination with the current network state S, selects the action corresponding to the maximum value of the multiplication result of the current preference and the Q vector in the optional action space of each task with a probability of ε, and selects a random action with a probability of 1-ε: The objective function is set up as follows: where γ is the discount factor, Indicates the action corresponding to the maximum value of the multiplication result of the preference and Q vector in the main network, θ is the parameter of the main network, and θ′ is the parameter of the target network; The loss function is: L(θ t )=E s,a,w [(y t -Q(s t ,a t ,w;θ t )) 2 ] In the above formula, Q represents the Q value obtained by the Q network, s′ is the next state output by the Q network, and the loss function value is used to update the Q network.

6. A method for offloading dependent task calculations that satisfies user random preferences according to claim 5, characterized in that: The specific steps of step 7 are: storing the system state, system action, immediate reward, preference of the current time slot and the system state of the next time slot obtained from the training in the experience pool, randomly sampling m samples from the sample pool from the experience buffer pool for training, minimizing the loss function according to the gradient descent method and updating the main network parameters. During the training process, the input of the network is the current state and preference, and the output is the Q vector; the training aims to maximize the reward and obtain the optimal unloading decision. After the iteration, the long-term cumulative reward reaches the optimal, and the optimal unloading decision can be obtained.

Images