LEO satellite network routing computation unloading method based on MAPPO

Abstract

The invention provides an LEO satellite network routing calculation unloading method based on MAPPO, and belongs to the technical field of network routing and calculation unloading, and the method comprises the steps: S1, constructing an LEO satellite network with inter-satellite routing and satellite-ground unloading functions at the same time, and obtaining a first mathematical expression of an optimization target based on the LEO satellite network; s2, optimizing the first mathematical expression of the optimization target to obtain a second mathematical expression of the optimization target; s3, taking each satellite user generating the image calculation task as an intelligent agent, and modeling as a Markov decision process; and S4, solving a second expression of the optimization target by using MAPPO deep reinforcement learning, and updating Actor neural network parameters and Critic neural network parameters in a continuous interaction process with the environment so as to obtain a strategy with optimal performance. According to the method, the total time delay of the network is minimum under the condition that the constraint of cache region capacity, link load and calculation time delay is met.

Description

Technical Field

[0001] The invention belongs to the technical field of network routing and calculation unloading, and in particular relates to a LEO satellite network routing calculation unloading method based on MAPPO. Background Art

[0002] Ground communication technology faces the challenge of difficult coverage in areas such as deserts, oceans, and mountains where infrastructure is underdeveloped. LEO satellite communications can meet the communication needs of most parts of the world, and therefore have attracted the interest of more and more researchers. LEO satellite constellation technology represented by Starlink can provide high-quality communication services to global communication users at a lower cost, and plays an important role in the field of meteorological monitoring.

[0003] In order to complete the meteorological monitoring task, a large amount of image data needs to be transmitted and calculated. In addition, the long electromagnetic wave propagation distance of the inter-satellite link makes the local computing task method increasingly unable to meet the real-time communication requirements of users under large-scale loads. Usually, the image computing task will be offloaded to other nodes for calculation. However, the traditional inter-satellite offloading calculation method can only consider offloading to satellites in the line-of-sight area, which makes insufficient use of the computing resources of the LEO satellite network. At the same time, when the load changes frequently and causes the link weight to change dynamically, the existing shortest path routing algorithm based on static weights will not be able to be used. In addition, existing research has not jointly considered the LEO satellite network routing and computing offloading problems, and the overall delay is too high. Summary of the invention

[0004] The purpose of the present invention is to propose a LEO satellite network routing calculation offloading method based on MAPPO, which minimizes the total network delay while satisfying the constraints of buffer area capacity, link load and calculation delay.

[0005] The present invention is achieved through the following technical solutions:

[0006] A method for offloading LEO satellite network routing calculation based on MAPPO comprises the following steps:

[0007] Step S1, constructing a LEO satellite network with both inter-satellite routing and satellite-to-ground unloading functions, the LEO satellite network includes a satellite user cluster, an unloading computing satellite cluster, a relay satellite cluster and a base station satellite, and obtaining a first mathematical expression of an optimization target based on the LEO satellite network, the first mathematical expression being related to queuing delay, propagation delay, delay on a satellite unloading path, delay of a satellite computing task, load ratio of links included in the computing route, and selection of an unloading computing satellite;

[0008] Step S2, according to the source satellite set and the destination satellite set in the LEO satellite network remaining unchanged within a time slot, optimizing the first mathematical expression of the optimization target, and obtaining a second mathematical expression of the optimization target related to queuing delay, propagation delay, delay on the satellite offloading path, delay of the satellite computing task, and selection of the offloading computing satellite;

[0009] Step S3: each satellite user that generates an image computing task is regarded as an intelligent agent and modeled as a Markov decision process;

[0010] Step S4: randomly initialize the Actor neural network parameters and Critic neural network parameters of each agent, set the parameters, use MAPPO deep reinforcement learning to solve the second expression of the optimization objective, and update the Actor neural network parameters and Critic neural network parameters in the process of continuous interaction with the environment to obtain the strategy with optimal performance.

[0011] Furthermore, in step S1, the LEO satellite network includes multiple satellite nodes and multiple ground relay forwarding nodes for assisting unloading, the satellite nodes are divided into a satellite user cluster s, an unloading computing satellite cluster u, a relay satellite cluster m and a base station satellite d, and the satellite user cluster s = {s1, s2, s3, s4, ..., s K} includes K LEO satellite users that can generate meteorological image computing tasks, offloading computing satellite cluster u={u1,u2,u3,u4,…,u N} includes N LEO offloading computing satellites that provide image computing services, and the relay satellite cluster m = {m1,m2,m3,m4,…,m L The cluster includes L LEO relay satellites that are only responsible for relaying. The meteorological image computing tasks of various places taken by the LEO satellite cluster s are routed between satellites and sent to the LEO satellite cluster u that provides image computing services for computing. The computed image computing tasks are then routed between satellites and sent to the base station satellite d.

[0012] Furthermore, the step S1 specifically includes the following steps:

[0013] Step S11, initialize the image calculation task of each LEO satellite user: In each time slot, each LEO satellite user s k ∈s has an indivisible computationally intensive task, and the data stream of each task is provided by the LEO satellite user s k ∈s is composed of a series of independent data streams that obey Poisson distribution sent by users in the service area;

[0014] Step S12: Initialize the LEO satellite network routing process: calculate the queuing delays of satellite node i and satellite node j respectively. and propagation delay

[0015] Step S13: Initialize the LEO satellite network calculation unloading process: calculate the delay on the satellite unloading path and satellite computing task latency Get the satellite's unloading calculation delay

[0016] Step S14: Obtain the first mathematical expression of the optimization target: in, represents the time slot t from the source satellite user s k Each link in the calculated route to the destination base station satellite d The load ratio, represents the time slot t from the source satellite user s k Calculate route to destination base station satellite d All links contained in link i,j A collection of It represents the image computing task along the source satellite user s in time slot t k Calculate route to destination base station satellite d The total delay of quene i (t) represents the length of the packet queue on satellite node i in time slot t, quene max (t) represents the maximum length of the packet queue that can be cached on satellite node i in time slot t, represents the source satellite user s in time slot t k The generated computing task flow, size d (t) represents the amount of computing task data received by base station satellite d in time slot t, represents the delay threshold, N() represents the adjacent nodes, and V0 represents the set of all source satellite user nodes and destination base station satellite nodes in the data stream of all image computing tasks.

[0017] Furthermore, in step S12, the queuing delay According to the formula Calculate the propagation delay According to the formula Calculate, where orb(i) represents the orbital number of satellite node i, orb(j) represents the orbital number of satellite node j, C i,j (t) represents the channel capacity between satellite node i and satellite node j in time slot t, C i,g (t) represents the channel capacity between satellite node i and ground relay node g in time slot t, C g,j(t) represents the channel capacity between the ground relay node g and the satellite node j in time slot t, d i,j (t) represents the distance between satellite node i and satellite node j, d i,g (t) represents the distance between satellite node i and ground relay node g, d g,j (t) represents the distance between the ground relay node g and the satellite node j, and c represents the speed of electromagnetic wave propagation.

[0018] Furthermore, in step S2, the second mathematical expression of the optimization objective is:

[0019]

[0020] Furthermore, the step S3 specifically includes the following steps:

[0021] Step S31: The state space of each agent in each time slot consists of three parts: the number of CPU computing resources allocated to the offload computing satellite selected for the computing task and The distance from the source satellite user to the destination base station satellite routing path and And the average throughput of the satellite routing path from the source satellite user to the destination base station

[0022] Step S32: Each agent selects to unload the computing satellite as an action based on the state obtained by observing the environment. k (t);

[0023] Step S33: Each agent calculates the latency of the computational task routing calculation unloading generated by itself and sets the reward function r k (t).

[0024] Furthermore, the step S4 specifically includes the following steps:

[0025] Step S41: Randomly initialize the Actor neural network parameters output by each agent and Critic neural network parameters And set the parameters, specifically: clear the experience replay pool G, set the activation function type ζ k , the number of neurons ψ k , sampling size g m , update the number of parameter sets k e , learning rate l r , Network parameter optimizer type o p , maximum number of training sets Steps per episode

[0026] Step S42: Number of training sets teps Iterate step by step from 0 to the maximum number of training steps At the beginning of each training episode, randomly select the action set action (0) (t), reset the state set state (0) (t), where represents the action set of all K agents at the zth iteration in each training episode, and is expressed as Represents the state set of all K agents at the zth iteration in each training episode, where z is an integer;

[0027] Step S42: Set the state (z) (t) is input into the Actor neural network to obtain the action category distribution set of K agents at the z+1th iteration. The action category distribution set of the kth agent at the z+1th iteration is Each agent follows the formula Select the action with the highest probability in the action category distribution, or randomly sample the actions in the action category distribution, and assign the action selected by each agent to Merge into a new action set action (z+1) (t) is combined, η is the probability of selecting the action with the highest probability in the action category distribution, and at the same time, the state set state (z) (t) is input into the Critic neural network to obtain the value set of K agents at the z+1th iteration. The value of the kth agent at the z+1th iteration is Represented, and according to the Markov modeling method of step S3, the state space set state at the (z+1)th iteration is obtained (z+1) (t) and reward r (z+1) (t), for reward r (z+1) (t) is normalized to obtain

[0028] Step S43: When the experience replay pool G is full, the new data stored in the experience replay pool G Overwrite the earlier stored in the experience replay pool G

[0029] Step S44: represents the strategy output by the Actor neural network of the kth agent at the zth iteration, represents the strategy output by the Critic neural network of the kth agent at the zth iteration. In order to avoid a large difference between the new strategy at the z+1th iteration and the old strategy at the zth iteration, the objective function of MAPPO deep reinforcement learning is as follows: Shearing is performed, where ε is the shearing coefficient, is the generalized advantage estimator, γ is the discount factor, E t () indicates to obtain the expectation, Indicates that The value of is clipped to [1-ε,1+ε];

[0030] Step S45: Setting the Critic neural network strategy The loss function is Actor Neural Network Strategy The loss function is in, represents the target value, By Status The input parameter to the network is α k The action category probability prob output by the Actor neural network k (t) is calculated, a1 and a2 are loss coefficients respectively;

[0031] Step S46: Calculate the Critic neural network strategy The loss function and Actor Neural Network Strategy The loss function The gradient of back propagation updates the parameters α of the Actor neural network and the Critic neural network k and β k ;

[0032] Step S47: In the continuous interaction with the environment, the parameters of the Actor neural network and the Critic neural network are continuously updated, and the action strategy output according to the state is continuously optimized. When the number of training sets reaches When , the interaction with the environment and the training of the agent are ended.

[0033] Furthermore, in step S13, the satellite unloading calculation delay is specifically Among them, the delay on the offloading path is Indicates the unloading of the calculation satellite u in time slot t k ∈u and the distance between the ground relay node g, Indicates the unloading of the calculation satellite u in time slot t k The channel capacity of the link between ∈u and the ground relay node g, represents the computing task data flow from the satellite user to the base station satellite in time slot t. The delay of the satellite computing task is in, Indicates unloading of computing satellite u kThe amount of task data that needs to be calculated on ∈u, Indicates that from satellite user s k ∈s is routed to the base station satellite d∈V o The data size of the image computing task is Represents the computing task data flow The calculation density of Indicates the maximum number of CPU computing resources that can be scheduled on the offload computing satellite.

[0034] Furthermore, in step S14, the total delay is expressed as

[0035] The present invention has the following beneficial effects:

[0036] 1. The present invention first obtains a first mathematical expression of the optimization target of considering queuing delay, propagation delay, delay on the satellite unloading path, delay of satellite computing task, satellite unloading computing delay, load ratio of links included in the computing route, and selection of unloading computing satellite. Then, considering that the source satellite set and the destination satellite set in the LEO satellite network remain unchanged within a time slot, the first mathematical expression is optimized to obtain a second mathematical expression related to queuing delay, propagation delay, delay on the satellite unloading path, delay of satellite computing task, and selection of unloading computing satellite. Then, each satellite user that generates an image computing task is taken as an intelligent agent and modeled as a Markov decision process. Finally, the Actor of each intelligent agent is randomly initialized. The neural network parameters and the Critic neural network parameters are set, and the MAPPO deep reinforcement learning is used to solve the second expression of the optimization target. The Actor neural network parameters and the Critic neural network parameters are updated in the process of continuous interaction with the environment to obtain the strategy with the best performance, so as to achieve the minimum total network delay under the constraints of buffer capacity, link load and computing delay, and reasonably allocate computing resources to complete the offload calculation of tasks when the parallel transmission load is high, and make full use of the computing resources in the LEO network to reduce the delay required to complete the computing task. Compared with the traditional heuristic optimization algorithm, the present invention can avoid falling into the local optimal solution and improve the adaptability of the optimization strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention is further described in detail below with reference to the accompanying drawings.

[0038] Figure 1 It is a flow chart of the present invention.

[0039] Figure 2 This is a system model diagram of the present invention.

[0040] Figure 3 This is a framework diagram of the MAPPO algorithm of the present invention.

[0041] Figure 4 This is a simulation diagram of the relationship between the total training reward and the number of training episodes for satellite users in the present invention.

[0042] Figure 5 In this invention, when the learning rate is set to 10 -4 Result diagram of each source satellite user network routing and calculation satellite selection when .

[0043] Figure 6 In this invention, when the learning rate is set to 10 -4 Simulation diagram of the relationship between the training reward of each agent and the number of training episodes.

[0044] Figure 7 This is a simulation diagram of the relationship between the total delay of satellite users and the number of training sets in the present invention.

[0045] Figure 8 In this invention, when the learning rate is set to 10 -4 Simulation diagram of the relationship between the delay of each agent and the number of training episodes. DETAILED DESCRIPTION

[0046] like Figure 1 As shown, the MAPPO-based LEO satellite network routing calculation offloading method includes the following steps:

[0047] Step S1, constructing a LEO satellite network with both inter-satellite routing and satellite-to-ground unloading functions, the LEO satellite network includes a satellite user cluster, an unloading computing satellite cluster, a relay satellite cluster and a base station satellite, and obtaining a first mathematical expression of an optimization target based on the LEO satellite network, the first mathematical expression being related to queuing delay, propagation delay, delay on a satellite unloading path, delay of a satellite computing task, load ratio of links included in the computing route, and selection of an unloading computing satellite;

[0048] Specifically, Figure 2 As shown in FIG. 1 , the LEO satellite network includes multiple satellite nodes and multiple ground relay forwarding nodes for assisting unloading. The satellite nodes are divided into a satellite user cluster s, an unloading computing satellite cluster u, a relay satellite cluster m, and a base station satellite d. The satellite user cluster s = {s1, s2, s3, s4, …, s K} includes K = 14 LEO satellite users that can generate meteorological image computing tasks, offloading computing satellite cluster u = {u1,u2,u3,u4,…,u N} includes N = 7 LEO offloading computing satellites providing image computing services, and the relay satellite cluster m = {m1,m2,m3,m4,…,m LThe cluster includes L = 7 LEO relay satellites that are only responsible for relaying. These satellites are divided into 3 orbits, each with 5 satellites. The orbital altitude set is [200, 300, 400] km, corresponding to the orbital altitude of each orbit. The meteorological image computing tasks of various places taken by the LEO satellite cluster s are routed to the LEO satellite cluster u that provides image computing services through inter-satellite routing and calculated. The calculated image computing tasks are routed to the base station satellite d through inter-satellite routing. Each satellite user s k ∈s generated by the image computing task cannot be split, each satellite user s k The image computing task of ∈s can only be offloaded on one satellite and finally routed to the base station satellite.

[0049] Step S1 specifically includes the following steps:

[0050] Step S11: Initialize each LEO satellite user s k ∈s image computing task: Since the image computing task offloading in the LEO satellite network runs in a time slot-by-time slot manner, the duration of each time interval is T c , and the time slots can be indexed by α=0,1,...,T; in each time slot, each LEO satellite user s k ∈s has an indivisible computationally intensive task and needs to select a satellite u from the offload computing satellite cluster u that can provide the computing task service. k , offloading the image computing task to the satellite u k ∈u; in time slot t, satellite user s k ∈s generated by the computation task can be represented by a tuple Indicates that represents satellite users s k ∈s is the amount of data generated by the image computing task, in bits; It represents the computing density, which measures how much CPU computing resources are needed to complete the calculation of each bit of image computing task. The unit is cycle / bit. There are a total of K parallel source node to destination node tasks in the network. The data flow of all image computing tasks is Indicates that the data stream of each mission is composed of the LEO satellite user s k ∈s is composed of a series of independent data streams sent by users in the service area and obeying Poisson distribution, which can be expressed as t=0,1,…T, where n is the service area to the source satellite s in time slot t k ∈s is the total number of users sending data, Indicates that user i moves to The amount of data sent is subject to the mean Poisson distribution; Since Poisson distribution is additive, The mean is Poisson distribution, using In this embodiment, The value is 10 4 .

[0051] Step S12: Initialize the LEO satellite network routing process: calculate the queuing delays of satellite node i and satellite node j respectively. and propagation delay

[0052] Specifically, according to Shannon's theory, the channel capacity between satellite node i and satellite node j in time slot t is:

[0053] Where P t , G t , G r are respectively the transmitting power, transmitting gain and receiving gain, λ is the wavelength, σ 2 is the power of additive white Gaussian noise, B is the link bandwidth, and specifically in this embodiment, P t The value is 20W, G t The value is 34dB, G r The value of is 37.5dB, the value of λ is 0.08m, and the value of σ 2 The value of A is 1W and the value of B is 100MHz.

[0054] The satellite node first stores the received data packet in the buffer area, and sends the stored data packet to the next hop satellite node after receiving the request to transmit data. The link link between satellite node i and satellite node j in time slot t in the same orbit i,j The queue delay on It is expressed as: orb(i) represents the orbital number of satellite node i, orb(j) represents the orbital number of satellite node j, C i,j (t) represents the channel capacity between satellite node i and satellite node j in time slot t, represents the packet queue length of the buffer area on satellite node i in time slot t, which shows that any Nodes in The length of the upper packet queue quene i (t) is determined by all data flows passing through satellite node i, where V o Represents the data flow of all image computing tasks The set of all source satellite user nodes and destination base station satellite nodes in, Indicates the source satellite user s k∈s is the set of nodes passed through in the satellite routing process to the destination base station.

[0055] The transmission delay from satellite node i to satellite node j in time slot t within the same orbit is expressed as Among them, d i,j (t) represents the distance between satellite node i and satellite node j, c = 3 × 10 8 m / s represents the speed at which electromagnetic waves propagate;

[0056] In order to avoid the problem of difficult alignment of phased array antennas, ground relay nodes are used to assist the communication between satellite nodes in different orbits. Therefore, the link link between satellite node i and satellite node j in different orbits i,j The queue delay on It includes the downlink queuing delay from satellite node i to ground relay node g and the uplink queuing delay from ground relay node g to satellite node j, that is, Among them, C i,g (t) represents the channel capacity between satellite node i and satellite node g in time slot t, C g,j (t) represents the channel capacity between satellite node g and satellite node j in time slot t.

[0057] At the same time, the transmission delay from satellite node i to satellite node j on different orbits in time slot t includes the downlink propagation delay from satellite node i to ground relay node g and the uplink propagation delay from ground relay node g to satellite node j, which is given by the following formula: Among them, d i,g (t) represents the distance between satellite node i and ground relay node g, d g,j (t) represents the distance between the ground relay node g and the satellite node j.

[0058] Step S13: Initialize the LEO satellite network calculation unloading process: calculate the delay on the satellite unloading path and satellite computing task latency Get the satellite's unloading calculation delay

[0059] More specifically, assuming that each LEO satellite u k ∈u has different computing capabilities The unit is CPU cycle / s. Specifically in this embodiment, the computing power of the offload computing satellite is randomly distributed between [2,10]Gcycles / s. k The amount of task data being calculated simultaneously is given by Indicates that, in time slot t, from source satellite user s k ∈s to the destination base station satellite d∈V o Routing Indicates that the satellite u is calculated k It is necessary to send the image computing task to the ground relay forwarding node in the service area through the satellite-to-ground link, and call the computing resources of the ground relay forwarding node to assist the satellite u k Calculation task, then calculate satellite u k The total delay required to offload the task and complete the computation at time slot t is Among them, the delay on the offloading path is Indicates the unloading of the calculation satellite u in time slot t k ∈u and the distance between the ground relay node g, Indicates the unloading of the calculation satellite u in time slot t k The channel capacity of the link between ∈u and the ground relay node g, represents the computing task data flow from the satellite user to the base station satellite in time slot t, It indicates that the ground relay node g has processed the source satellite user s in time slot t. k ∈s to the destination base station satellite d∈V o The computing task data flow The data size of the post-task, λ is the data compression ratio, indicating the degree to which the task data stream is compressed during the calculation process. Specifically in this embodiment, the value of λ is 0.1.

[0060] At the same time, the CPU computing resources of the ground-based auxiliary offloading computing node g are called to perform image computing tasks. The delay of calculation (i.e. the delay of satellite calculation task) is expressed as in, Indicates unloading of computing satellite u k The amount of task data that needs to be calculated on ∈u, Represents the computing task data flow The calculation density of Indicates the maximum number of CPU computing resources that can be scheduled on the offload computing satellite. Formula It shows that, assuming the image computing task Offloaded to satellite u k The computing resources allocated depend on the amount of data in the computing task itself. The weighted allocation of satellite u is based on the amount of data. k CPU computing resources on Give The computational tasks in .

[0061] Step S14: Obtain the first mathematical expression of the optimization target: This formula shows that the optimization goal is to minimize the maximum total task delay by reasonably allocating the set of computing paths of tasks under relevant constraints, where represents the time slot t from the source satellite user s k Each link in the calculated route to the destination base station satellite d The load ratio is 0 or 1, indicating that the link from satellite node i to satellite node j Whether it is activated; represents the time slot t from the source satellite user s k Calculate route to destination base station satellite d All links contained in link i,j A collection of It represents the image computing task along the source satellite user s in time slot t k Calculate route to destination base station satellite d The total delay of quene i (t) represents the length of the packet queue on satellite node i in time slot t, quene max (t) represents the maximum length of the packet queue that can be cached on satellite node i in time slot t, represents the source satellite user s in time slot t k The generated computing task flow, size d (t) represents the amount of computing task data received by base station satellite d in time slot t, represents the delay threshold, N() represents the adjacent nodes, and V0 is the base station satellite cluster.

[0062] Specifically, the optimization goal is to reasonably allocate tasks under relevant constraints. The calculation path set Minimize the largest task Delay.

[0063] In the first expression of the optimization objective, the first constraint indicates that the buffer queue length of each satellite node cannot exceed the maximum capacity of its own buffer, thereby avoiding the problem of image computing tasks being lost due to inability to store. The second constraint indicates the computing task flow generated by each source satellite user. The sum of the data volumes must be greater than or equal to the amount of computing task data received by the destination base station satellite. The third constraint indicates that since the source satellite user node generates image computing tasks itself, the amount of image computing task data flowing out of itself is greater than the amount of image computing task data flowing into itself. The fourth constraint indicates that for the relay satellite m that does not generate additional image computing tasks, k ∈m and offload computation to satellite u k∈u, the amount of data flowing into itself is greater than or equal to the amount of data flowing out of itself. The fifth constraint indicates that since the destination base station satellite does not generate image computing tasks, the amount of image computing task data flowing out of itself is less than the amount of image computing task data flowing into itself. The sixth constraint indicates that the delay of ground-based auxiliary satellite unloading must be lower than the corresponding delay threshold of the network, otherwise the non-local unloading calculation method considered in the present invention will be meaningless.

[0064] Step S2, according to the source satellite set and the destination satellite set in the LEO satellite network remaining unchanged within a time slot, optimizing the first mathematical expression of the optimization target to obtain a second mathematical expression related to queuing delay, propagation delay, delay on the satellite offloading path, delay of the satellite computing task, and selection of the offloading computing satellite;

[0065] Specifically, the first expression of the optimization objective is decoupled and converted into and unloaded computing node u k ∈u selects the relevant form; after determining each computing task flow The corresponding offload computing node u k ∈u, we only need to solve the problem of k ∈s to the destination base station satellite d∈V o Load distribution method Due to the offloading of computing node u k ∈u is already a constant, so we need to solve Just need the formula As shown in the figure, Dijkstra algorithm is used to independently solve the source satellite user s k ∈s to offload computation to satellite u k ∈u and offload computation to satellite u k ∈u to the destination base station satellite d∈V o Two-stage load distribution

[0066] The above analysis shows that the load distribution method and uninstall compute node u k ∈u, so we can first replace the optimization variables in the first expression Rewrite as Since the source satellite user s in time slot t k ∈s to the destination base station satellite d∈V o The calculated routing set From the source satellite user s in time slot t k ∈s to the destination base station satellite d∈V o Load distribution method of routing path and the selection of offloading calculation satellites, so the optimization variables are further rewritten as Then rewrite the first expression of the optimization objective into the following second expression: Convert the optimization goal into placing and unloading computing nodes u k ∈u selects the relevant form.

[0067] Step S3: each satellite user that generates an image computing task is regarded as an intelligent agent and modeled as a Markov decision process;

[0068] The specific steps include:

[0069] Step S31: The state space of each agent in each time slot consists of three parts: computing tasks The number of CPU computing resources allocated to the selected offload computing satellite From source satellite users k ∈s to the destination base station satellite d∈V o Routing Path The distance and and from the source satellite users k ∈s to the destination base station satellite d∈V o Average throughput of the routing path Right now Among them, the calculation task The number of CPU computing resources allocated to the selected offload computing satellite Expressed as In this formula, Indicates the unloading calculation satellite u in time slot t k The amount of image computing task data that needs to be calculated on ∈u, Represents the source satellite user s in time slot t k ∈s is routed to the destination base station satellite d∈V o The data size of the image computing task is Indicates the unloading calculation satellite u in time slot t k The maximum number of CPU computing resources that can be scheduled on ∈u.

[0070] At the same time, assuming that Indicates link The degree to which the amount of data transmitted by the computing task is compressed, the routing path The average throughput can be expressed as

[0071]

[0072] Finally, suppose represents the communication link from satellite i to satellite j Distance, routing path The sum of the distances can be expressed as

[0073] Step S32: Each agent selects to unload the computing satellite as an action based on the state obtained by observing the environment. k (t);

[0074] Specifically, in time slot t, each agent has a state k (t) Select the action to be performed k (t), select the task in time slot t The corresponding unloading calculation satellite is used Indicates the number of satellites that are offloaded. The action selection value is The integers between correspond to the LEO offload computing satellite numbers that provide image computing services, specifically expressed as

[0075] Step S33: Each agent calculates the latency of the computational task routing calculation unloading generated by itself and sets the reward function

[0076] Step S4, randomly initialize the Actor neural network parameters and the Critic neural network parameters of each agent, set the parameters, use MAPPO deep reinforcement learning to solve the second expression of the optimization target, and update the Actor neural network parameters and the Critic neural network parameters in the process of continuous interaction with the environment to obtain the strategy with the best performance;

[0077] The specific steps include:

[0078] Step S41: represents the source satellite user s k ∈s, the Actor neural network parameters output by the agent, represents the source satellite user s k ∈s, the Critic neural network parameters output by the agent are first randomly initialized to the Actor neural network parameters corresponding to each agent and Critic neural network parameters According to the scale of the LEO satellite network and the complexity of the Markov problem constructed in step S3, the activation function type ζ of each agent is set. k , the number of neurons ψ k , specifically in this implementation, k is the TANH function, ψ k The value of is 10;

[0079] Then, clear the experience replay pool G with capacity g = 40 and set the sampling size g m =8, Update parameter set number k e=15, learning rate l r =10 -4 , Network parameter optimizer type o p (specifically the Adam optimizer), maximum number of training sets Steps per episode

[0080] Step S42: Figure 3 As shown, MAPPO multi-intelligence deep reinforcement learning is used to solve the second expression of the optimization objective, and the number of training sets is t eps Iterate step by step from 0 to the maximum number of training steps At the beginning of each training episode, randomly select the action set action (0) (t), reset the state set state (0) (t), where represents the action set of all K agents at the zth iteration in each training episode, and is expressed as Represents the state set of all K agents at the zth iteration in each training episode, where z is an integer. At the beginning of each training episode, the action set action needs to be randomly selected (0) (t), reset the state set state (0) (t);

[0081] Step S42: Set the state (z) (t) is input into the Actor neural network, and the action category distribution set of K agents at the z+1th iteration is obtained through the input layer, hidden layer and output layer. The action category distribution set of the kth agent at the z+1th iteration is Each agent follows the formula Select the action with the highest probability in the action category distribution, or randomly sample the actions in the action category distribution, and assign the action selected by each agent to Merge into a new action set action (z+1) (t) is combined, η is the probability of selecting the action with the highest probability in the action category distribution, and at the same time, the state set state (z) (t) is input into the Critic neural network, and the value set of K agents at the z+1th iteration is obtained through the input layer, hidden layer and output layer. Represents the output action of the Actor network The evaluation of the kth agent at the z+1th iteration is express;

[0082] Output a new action set action in the Actor network (z+1)(t), the state space set state at the (z+1)th iteration is obtained according to the Markov modeling method in step S3 (z+1) (t) and reward r (z+1) (t), among which, the award In order to improve the stability of training, the reward r (z+1) (t) is normalized to obtain Among them, min(r (z+1) (t)) represents the reward set r (z+1) (t), the minimum value in max(r (z+1) (t)) represents the reward set r (z+1) (t) is the maximum value. The state set state obtained by the above process (z+1) (t), action set action (z+1) (t), value set value (z+1) (t), normalized reward set Aggregate into a collection And store it in the experience replay pool G. Repeat the above process times, making eps step Step by step from 0 to

[0083] Step S43: The training of each episode above gradually increases the size of the experience replay pool G. When the experience replay pool G is filled, the parameters of the Actor network and the Critic network are updated according to the gradient back propagation of the loss function. Overwrite the earlier stored in the experience replay pool G

[0084] Step S44: represents the strategy output by the Actor neural network of the kth agent at the zth iteration, represents the strategy output by the Critic neural network of the kth agent at the zth iteration. In order to avoid a large difference between the new strategy at the z+1th iteration and the old strategy at the zth iteration, the objective function of MAPPO deep reinforcement learning is as follows: Shearing is performed, where ε is the shearing coefficient, is the generalized advantage estimator, γ=0.99 is the discount factor, E t [] indicates the expectation. Indicates that The value of is clipped to [1-ε,1+ε];

[0085] Step S45: Setting the Critic neural network strategy The loss function is Actor Neural Network Strategy The loss function is in, represents the target value, By Status The input parameter to the network is α k The action category probability prob output by the Actor neural network k (t) is calculated, a1 = 1 and a2 = 0.01 are loss coefficients respectively;

[0086] Step S46: Calculate the Critic neural network strategy The loss function and Actor Neural Network Strategy The loss function The gradient of back propagation updates the parameters α of the Actor neural network and the Critic neural network k and β k ;

[0087] Step S47: In the continuous interaction with the environment, the parameters of the Actor neural network and the Critic neural network are continuously updated, and the action strategy output according to the state is continuously optimized. When the number of training sets reaches t max hour,

[0088] EPS ends the interaction with the environment and the training of the intelligent agent.

[0089] Figure 4 In the simulation, the total training rewards of three training learning rates are compared: 1) the training learning rate is set to 10 -4 ; 2) The training learning rate is set to 10 -3 ; 3) The training learning rate is set to 10 -2 ;Depend on Figure 4 It can be seen that the learning rate is set to 10 -4 The training reward converges very smoothly when the learning rate is set to 10 -3 The training reward convergence fluctuates, and the learning rate is set to 10 -2 The training reward convergence fluctuates greatly. In addition, the learning rate is set to 10 -4 The training reward converges the slowest, and the learning rate is set to 10 -3 The training reward converges the second fastest, and the learning rate is set to 10 -2 The training reward converges fastest. The above simulation results show that the convergence speed of the three learning rate settings is not much different, but the stability of the convergence curve is quite different. Therefore, the learning rate is set to 10 -4 The training results are better.

[0090] Figure 5 It shows 14 parallel transmission routes to the destination satellite and the offloaded computing satellite selected by the task on each route, where the minimum hop number is 1 and the maximum hop number is 5. Nodes with stronger computing power undertake more computing tasks, while nodes with weaker computing power undertake fewer computing tasks.

[0091] Figure 6 The horizontal axis is the number of training sets, ranging from 0 to 3000, and the vertical axis is the training reward of each agent in the satellite user offloading calculation. The simulation experiment compares and analyzes the training rewards of 14 agents in the network. Figure 6 It can be seen that the training rewards of all agents converged almost at the same time, the training reward of agent number 3 was the largest when it converged, and the training reward of agent number 12 was the smallest when it converged. This is because the number of routing hops of agent number 3 is smaller than that of agent number 12, and it can be allocated more CPU computing resources than agent number 12.

[0092] Figure 7 The horizontal axis is the number of training sets, ranging from 0 to 3000, and the vertical axis is the total delay of satellite user offloading calculation. The simulation experiment conducted a comparative analysis of the total delay of three training learning rates: 1) The training learning rate is set to 10 -4 ; 2) The training learning rate is set to 10 -3 ; 3) The training learning rate is set to 10 -2 ;Depend on Figure 7 It can be seen that the learning rate is set to 10 -4 The total delay converges very smoothly, and the learning rate is set to 10 -3 The total delay convergence fluctuates, and the learning rate is set to 10 -2 The total delay convergence fluctuates greatly. In addition, the learning rate is set to 10 -4 The total delay convergence speed is the slowest, and the learning rate is set to 10 -3 The total delay converges the second fastest, and the learning rate is set to 10 -3 The above simulation results show that the total delay convergence speed of the three learning rate settings is not much different, but the stability of the total delay convergence curve is quite different. Therefore, the learning rate is set to 10 -4 The training results are better.

[0093] Figure 8 The horizontal axis is the number of training sets, ranging from 0 to 3000, and the vertical axis is the delay of satellite users offloading the calculation of each agent. The simulation experiment compares and analyzes the training rewards of 14 agents in the network. Figure 8It can be seen that the training rewards of all agents converged almost at the same time, the latency of agent number 3 converged the lowest, and the latency of agent number 12 converged the highest. This is because the routing process of agent number 3 requires shorter transmission latency and queuing latency than the routing process of agent number 12, and the computation offloading process requires shorter offloading computation latency.

[0094] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A method for offloading LEO satellite network routing calculation based on MAPPO, characterized in that: The steps include: Step S1, constructing a LEO satellite network with both inter-satellite routing and satellite-to-ground unloading functions, the LEO satellite network includes a satellite user cluster, an unloading computing satellite cluster, a relay satellite cluster and a base station satellite, and obtaining a first mathematical expression of an optimization target based on the LEO satellite network, the first mathematical expression being related to queuing delay, propagation delay, delay on a satellite unloading path, delay of a satellite computing task, load ratio of links included in the computing route, and selection of an unloading computing satellite; Step S2, according to the source satellite set and the destination satellite set in the LEO satellite network remaining unchanged within a time slot, optimizing the first mathematical expression of the optimization target, and obtaining a second mathematical expression of the optimization target related to queuing delay, propagation delay, delay on the satellite offloading path, delay of the satellite computing task, and selection of the offloading computing satellite; Step S3: each satellite user that generates an image computing task is regarded as an intelligent agent and modeled as a Markov decision process; Step S4: randomly initialize the Actor neural network parameters and Critic neural network parameters of each agent, set the parameters, use MAPPO deep reinforcement learning to solve the second expression of the optimization objective, and update the Actor neural network parameters and Critic neural network parameters in the process of continuous interaction with the environment to obtain the strategy with optimal performance.

2. The method for offloading LEO satellite network routing calculation based on MAPPO according to claim 1, characterized in that: In step S1, the LEO satellite network includes multiple satellite nodes and multiple ground relay forwarding nodes for auxiliary unloading, the satellite nodes are divided into a satellite user cluster s, an unloading computing satellite cluster u, a relay satellite cluster m and a base station satellite d, and the satellite user cluster s = {s1, s2, s3, s4, ..., s K } includes K LEO satellite users that can generate meteorological image computing tasks, offloading computing satellite cluster u={u1,u2,u3,u4,…,u N } includes N LEO offloading computing satellites that provide image computing services, and the relay satellite cluster m = {m1,m2,m3,m4,…,m L The cluster includes L LEO relay satellites that are only responsible for relaying. The meteorological image computing tasks of various places taken by the LEO satellite cluster s are routed between satellites and sent to the LEO satellite cluster u that provides image computing services for computing. The computed image computing tasks are then routed between satellites and sent to the base station satellite d.

3. The method for offloading LEO satellite network routing calculation based on MAPPO according to claim 2, characterized in that: The step S1 specifically includes the following steps: Step S11, initialize the image calculation task of each LEO satellite user: In each time slot, each LEO satellite user s k ∈s has an indivisible computationally intensive task, and the data stream of each task is provided by the LEO satellite user s k ∈s is composed of a series of independent data streams that obey Poisson distribution sent by users in the service area; Step S12: Initialize the LEO satellite network routing process: calculate the queuing delays of satellite node i and satellite node j respectively. and propagation delay Step S13, initializing the LEO satellite network calculation unloading process: by calculating the delay on the satellite unloading path and satellite computing task latency Get the satellite's unloading calculation delay Step S14: Obtain the first mathematical expression of the optimization target: in, represents the time slot t from the source satellite user s k Each link in the calculated route to the destination base station satellite d The load ratio, represents the time slot t from the source satellite user s k Calculate route to destination base station satellite d All links contained in link i,j A collection of It represents the image computing task along the source satellite user s in time slot t k Calculate route to destination base station satellite d The total delay of quene i (t) represents the length of the packet queue on satellite node i in time slot t, quene max (t) represents the maximum length of the packet queue that can be cached on satellite node i in time slot t, represents the source satellite user s in time slot t k The generated computing task flow, size d (t) represents the amount of computing task data received by base station satellite d in time slot t, represents the delay threshold, N() represents the adjacent nodes, and V0 represents the set of all source satellite user nodes and destination base station satellite nodes in the data stream of all image computing tasks.

4. The method for offloading LEO satellite network routing calculation based on MAPPO according to claim 3, characterized in that: In step S12, the queuing delay According to the formula Calculate the propagation delay According to the formula Calculate, where orb(i) represents the orbital number of satellite node i, orb(j) represents the orbital number of satellite node j, C i,j (t) represents the channel capacity between satellite node i and satellite node j in time slot t, C i,g (t) represents the channel capacity between satellite node i and ground relay node g in time slot t, C g,j (t) represents the channel capacity between satellite ground relay node g and satellite node j in time slot t, d i,j (t) represents the distance between satellite node i and satellite node j, d i,g (t) represents the distance between satellite node i and satellite node g, d g,j (t) represents the distance between satellite node g and satellite node j, and c represents the speed of electromagnetic wave propagation.

5. The method for offloading LEO satellite network routing calculation based on MAPPO according to claim 4, characterized in that: In step S2, the second mathematical expression of the optimization target is:

6. The method for offloading LEO satellite network routing calculation based on MAPPO according to claim 5, characterized in that: The step S3 specifically includes the following steps: Step S31: The state space of each agent in each time slot consists of three parts: the number of CPU computing resources allocated to the offload computing satellite selected for the computing task The distance from the source satellite user to the destination base station satellite routing path and And the average throughput of the satellite routing path from the source satellite user to the destination base station Step S32: Each agent selects to unload the computing satellite as an action based on the state obtained by observing the environment. k (t); Step S33: Each agent calculates the latency of the computational task routing calculation unloading generated by itself and sets the reward function r k (t).

7. The method for offloading LEO satellite network routing calculation based on MAPPO according to claim 6, characterized in that: The step S4 specifically includes the following steps: Step S41: Randomly initialize the Actor neural network parameters output by each agent and Critic neural network parameters And set the parameters, specifically: clear the experience replay pool G, set the activation function type ζ k , the number of neurons ψ k , sampling size g m , update the number of parameter sets k e , learning rate l r , Network parameter optimizer type o p , maximum number of training sets Steps per episode Step S42: Number of training sets t eps Iterate step by step from 0 to the maximum number of training steps At the beginning of each training episode, randomly select the action set action (0) (t), reset the state set state (0) (t), where represents the action set of all K agents at the zth iteration in each training episode, and is expressed as represents the state set of all K agents at the zth iteration in each training episode, where z is an integer; Step S42: Set the state (z) (t) is input into the Actor neural network to obtain the action category distribution set of K agents at the z+1th iteration. The action category distribution set of the kth agent at the z+1th iteration is Each agent follows the formula Select the action with the highest probability in the action category distribution, or randomly sample the actions in the action category distribution, and assign the action selected by each agent to Merge into a new action set action (z+1) (t) is combined, η is the probability of selecting the action with the highest probability in the action category distribution, and at the same time, the state set state (z) (t) is input into the Critic neural network to obtain the value set of K agents at the z+1th iteration. The value of the kth agent at the z+1th iteration is Represented, and according to the Markov modeling method of step S3, the state space set state at the (z+1)th iteration is obtained (z+1) (t) and reward r (z+1) (t), for reward r (z+1) (t) is normalized to obtain Step S43: When the experience replay pool G is full, the new data stored in the experience replay pool G Overwrite the earlier stored in the experience replay pool G Step S44: represents the strategy output by the Actor neural network of the kth agent at the zth iteration, represents the strategy output by the Critic neural network of the kth agent at the zth iteration. In order to avoid a large difference between the new strategy at the z+1th iteration and the old strategy at the zth iteration, the objective function of MAPPO deep reinforcement learning is as follows: Shearing is performed, where ε is the shearing coefficient, is the generalized advantage estimator, γ is the discount factor, E t [] indicates seeking expectation, Indicates that The value of is clipped to [1-ε,1+ε]; Step S45: Setting the Critic neural network strategy The loss function is Actor Neural Network Strategy The loss function is in, represents the target value, By Status The input parameter to the network is α k The action category probability prob output by the Actor neural network k (t) is calculated, a1 and a2 are loss coefficients respectively; Step S46: Calculate the Critic neural network strategy The loss function and Actor Neural Network Strategy The loss function The gradient of back propagation updates the parameters α of the Actor neural network and the Critic neural network k and β k ; Step S47: In the continuous interaction with the environment, the parameters of the Actor neural network and the Critic neural network are continuously updated, and the action strategy output according to the state is continuously optimized. When the number of training sets reaches When , the interaction with the environment and the training of the agent are ended.

8. A MAPPO-based LEO satellite network routing calculation offloading method according to any one of claims 2 to 7, characterized in that: In step S13, the satellite unloading calculation delay is specifically: Among them, the delay on the offloading path is Indicates the unloading of the calculation satellite u in time slot t k ∈u and the distance between the ground relay node g, Indicates the unloading of the calculation satellite u in time slot t k The channel capacity of the link between ∈u and the ground relay node g, represents the computing task data flow from the satellite user to the base station satellite in time slot t. The delay of the satellite computing task is in, Indicates unloading of computing satellite u k The amount of task data that needs to be calculated on ∈u, Indicates that from satellite user s k ∈s is routed to the base station satellite d∈V o The data size of the image computing task is Represents the computing task data flow The calculation density of Indicates the maximum number of CPU computing resources that can be scheduled on the offload computing satellite.

9. A MAPPO-based LEO satellite network routing calculation offloading method according to any one of claims 3 to 7, characterized in that: In step S14, the total delay is expressed as

Images