Satellite-ground collaborative edge network resource allocation method based on depth deterministic strategy gradient

Abstract

The invention discloses a satellite-ground collaborative edge network resource allocation method based on depth deterministic strategy gradient, belongs to the field of network resource optimization in satellite internet, and adopts a DRL method to optimize allocation of bandwidth resources and computing resources. Firstly, a satellite-ground collaborative edge network resource allocation model containing a physical space and a digital twin space is designed. Secondly, establishing an allocation problem of communication resources and computing resources to balance the time delay and energy consumption of the system, and modeling a joint target optimization problem in the satellite-ground collaborative edge network as a Markov decision process (MDP); then, an algorithm based on a depth deterministic policy gradient (DDPG) is provided to intelligently decide allocation of bandwidth resources and calculate the allocation proportion of the resources, and system time delay and energy consumption are reduced to the maximum extent under the condition that constraints are met.

Description

Technical Field

[0001] The present invention relates to the field of satellite Internet technology, and in particular to a satellite-ground collaborative edge network resource allocation method based on deep reinforcement learning (DRL) technology. Background Art

[0003] As a supplement to ground networks, satellite networks can provide wide coverage and efficient transmission capabilities. In addition, satellite Internet supports secure data transmission and the need for privacy protection, providing a foundation for the implementation of various application scenarios such as the Internet of Things, smart cities and emerging technologies, and promoting the continuous advancement and innovation of global communication technology.

[0004] In the integrated space-ground information network, multiple satellite networks, ground networks and self-organizing networks are integrated through the satellite-ground collaborative network, especially for special scenarios such as remote areas, at sea, and disaster areas, which can provide users with ubiquitous and reliable network services. In particular, when devices perform delay-sensitive and computing-intensive tasks, it is an important trend to introduce edge computing technology in the satellite-ground collaborative network by drawing on the idea of ​​multi-access edge computing (MEC) in mobile networks. In the case of limited computing resources and energy for devices, MEC, as a technology that provides service environment and cloud computing capabilities at the edge of mobile networks, can solve the problem of computing resource shortage caused by huge business volume and real-time data processing. In the United States, MEC has become one of the potential technologies for expanding the capabilities of ground IoT devices. Deploying MEC in satellite networks can greatly reduce the task execution delay and energy consumption of ground IoT devices with limited computing power and battery capacity by offloading computing tasks to MEC servers on low-orbit satellites. On the other hand, with the increasing application of AI technology, it has become an inevitable trend to use algorithms and AI technology on satellites to realize intelligent services.

[0005] Based on the typical challenges faced by 5G networks, the digital twin network (DTN) that integrates DT technology with the Internet of Things, cloud computing, AI and other technologies has emerged. DTN uses real-time data collection, model building and intelligent decision-making to connect physical systems with corresponding digital models to achieve high interconnection and intelligent management. Unlike traditional network simulators, DTN can monitor and analyze the status and behavior of physical networks, and perform real-time simulation and optimization through digital models to provide intelligent support for decision-making. It provides new possibilities for improving efficiency, optimizing resource utilization and improving decision-making. Therefore, with the help of DT technology in satellite networks, the dynamic changes of satellite network nodes and links, real-time network spatiotemporal behavior and significantly different business types can be dynamically simulated to improve the digitalization and intelligence level of satellites.

[0006] The main obstacles to the full realization of cloud computing, MEC and IoT collaboration are in communication, computing, caching and control issues, especially the heterogeneity and complexity of IoT systems bring challenges to communication. Most of these problems are NP-hard problems, so by reducing the computational complexity, especially for large-scale IoT networks, heuristic algorithms such as genetic algorithms are widely used to solve these problems. However, the above algorithms are not suitable for real-time decision-making, because these problems often require quite a lot of iterations to find a satisfactory solution, and the development of DRL technology has great potential to solve these problems.

[0007] The present invention proposes a satellite-ground collaborative edge network resource optimization solution based on DRL technology, and uses digital twin technology to assist intelligent decision-making to minimize system latency and energy consumption under the condition of limited network resources. Summary of the invention

[0008] The present invention is different from the existing network resource allocation optimization scheme. It uses DRL technology to propose a satellite-ground collaborative edge network resource allocation method. First, a satellite-ground collaborative edge network resource allocation model is established. The DT technology is used to map the state behavior of the physical network into the satellite edge network digital space, and the current operating status of the network is monitored to assist decision-making. Secondly, the allocation problem of communication resources and computing resources is established to balance the system's latency and energy consumption, and the joint objective optimization problem in the satellite-ground collaborative edge network is modeled as a Markov decision process (MDP). A DDPG-based algorithm is proposed to intelligently decide the allocation of bandwidth resources and the allocation ratio of computing resources, so as to minimize the system latency and energy consumption while satisfying the constraints.

[0009] 1. Problems to be solved

[0010] For the satellite Internet environment, the traditional resource allocation algorithm does not fully consider the high dynamics of the new satellite network in communication and networking. When dealing with large-scale problems, the complexity of traditional solutions will increase rapidly, and it is difficult to handle dynamic changes and highly uncertain situations. Therefore, the present invention proposes a satellite-ground collaborative edge network resource allocation method based on DRL, and uses DT technology to simulate and optimize the physical network in real time to provide intelligent support for DRL technology to improve decision-making. DRL can not only achieve effective resource allocation by continuously trial and error to learn the best behavior strategy, but also make intelligent decisions to minimize latency and energy consumption in a complex satellite-ground network environment.

[0011] (II) Technical solution

[0012] The technical purpose of the present invention is to model the joint objective optimization problem as a Markov decision process (MDP) through DRL technology, and propose an algorithm based on deep deterministic gradient strategy (DDPG) to intelligently decide the allocation of bandwidth resources and the allocation ratio of computing resources, and use DT technology to monitor the current network status to assist decision-making, thereby minimizing system latency and energy consumption while satisfying constraints and improving the overall utility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Digital twin-assisted satellite-ground collaborative edge network system

[0014] Figure 2 MADRL algorithm framework based on DDPG

[0015] Figure 3 Actor-Critic Network Architecture DETAILED DESCRIPTION

[0016] The main process can be divided into the following two steps: establishing a satellite-ground collaborative edge network resource allocation model and designing a resource management solution based on the DDPG algorithm.

[0017] Step 1: Establish a satellite-ground collaborative edge network resource allocation model

[0018] Step 1.1: Design of physical space and digital twin space

[0019] As attached Figure 1 As shown, the system model proposed in the present invention includes physical entity space and digital twin space.

[0020] 1. Physical space

[0021] In physical space, adjacent satellites in different orbits will simultaneously cover a specific area on the ground. The UAVs in this area are grouped and associated with different satellites. Data is uploaded, models are sent, and real-time UAV status information and network topology updates are performed based on the association relationship. In addition, satellites use blockchain technology to propagate and aggregate models, and the aggregated global model is sent to UAVs. The system assumes that a collection of N identical UAVs is formed. These UAVs are grouped and associated with a set of M different edge servers: Assume that Indicates the association between UAVi and satellite j, that is, the data of UAVi is unloaded to satellite j. These satellites have corresponding computing capabilities and need to process the data unloaded by UAVs to train local models. At the same time, these satellites are regarded as blockchain nodes for model propagation and aggregation, and the aggregated global model is sent to its associated UAVs, thereby achieving secure and reliable data transmission.

[0022] 2. Digital Twin Space

[0023] In the digital twin space, DT technology is used to map the operating status of UAVs and satellite entities to the edge network digital space, and the operating status is used to simulate real UAVs and satellites. The real-time association status and network topology of satellites as edge servers and UAVs are also mapped to the digital space of edge servers. This study designed two DT models, namely DTs for UAVs and satellites. These DTs are built into the satellite digital space to ensure real-time status information mapping of UAVs equipment and satellites, and monitor the current operating status of the network. The system parameters are shown in Table 1.

[0024] (1) DT model for UAVs

[0025] The DT of UAVs is a digital representation of UAVs, which continuously interacts with entities and is updated in real time. For task k, the DT model of UAVi is represented as follows:

[0026]

[0027] (2) Satellite DT model

[0028] For satellite j, a similar expression can be given by:

[0029]

[0030] Step 1.2: Communication Model

[0031] 1. Satellite-to-ground user link

[0032] UAVs use multiplexing technology to transmit data to the satellite. The number of channels allocated to UAV device i is ξ i , which is one of the variables affecting system performance and is determined by the edge node. The transmission rate of UAV device i is as follows:

[0033]

[0034] 2. High-speed intersatellite links

[0035] For low-orbit satellite inter-satellite links, laser inter-satellite link technology can significantly improve the inter-satellite communication rate and adapt to the needs of broadband services. It has become the main development direction of inter-satellite high-speed communications.

[0036] Step 1.3: Delay Model

[0037] For task k, the total delay T of the system to complete global model training total Including communication delay T trans and calculation delay T com Considering the latency requirements of each UAV task, the total system latency from UAV uploading data to receiving the global model should be within T max within the range.

[0038] Communication delay includes uplink data transmission delay Model propagation delay T of intersatellite link inter And the model sending delay T of the downlink down During uplink data transmission, bandwidth allocation to UAVs directly affects system performance and resource utilization efficiency. Reasonable channel allocation can optimize network load balancing and avoid excessive resource occupation by individual devices, which may result in communication obstruction for other devices.

[0039] The computational delay mainly includes the processing delay T of the satellite for the data set of the associated UAVs. train and the processing delay of the block T block The satellite needs to allocate appropriate computing resources to the data sets of different UAVs and dynamically adjust the usage ratio of computing resources according to the load conditions.

[0040] Step 1.4: Energy Consumption Model

[0041] In the proposed system, transmission energy consumption and computing energy consumption are two key indicators for evaluating network performance. For the system proposed in this study, the total energy consumption E total Energy consumption of transmitting datasets containing UAVs to satellites And the hovering energy consumption of the whole process And the computing energy consumption of satellite training local models Considering the case of energy constraints, the total energy consumption cannot exceed the threshold E max .

[0042] After the UAVs upload data to the satellite, the satellite needs to allocate appropriate computing resources to different UAVs' tasks and dynamically adjust the usage ratio of computing resources according to the load. If too many computing resources are allocated, it will lead to an increase in computing energy consumption, and too few will lead to excessive computing delay. Therefore, the reasonable allocation of computing resources has an important impact on balancing the system's delay and energy consumption.

[0043] Step 1.5: Joint Optimization Problem Model

[0044] Under the condition of limited communication resources and computing resources, the algorithm needs to minimize latency and energy consumption and improve the overall utility of the system by reasonably allocating channel resources and computing resources during the training of the global model. Therefore, the resource optimization goal in the satellite-ground collaborative edge network can be defined as:

[0045]

[0046] Among them, variables ξ and ρ are decision variables, representing the allocation of bandwidth resources and the allocation ratio of satellite computing resources respectively. Constraint C1 represents the constraint on the weight coefficient, μ T and μ E They are the weight coefficients of system delay and energy consumption respectively. C2 means that for any satellite, the sum of computing resources allocated to its associated data must be less than its own total computing resources. C3 means that the total number of DTs in all satellites must be equal to the number of UAVs. C4 ensures that the data of any UAV is only uploaded to one satellite. C5 limits the allocation of bandwidth resources, indicating that the total number of channels allocated to UAVs must be less than the total number of channels. C6 and C7 ensure that the system delay and UAVs energy consumption are better than the set thresholds.

[0047] Step 2: Design of resource management solution based on DDPG algorithm

[0048] The above optimization problem (4) is an NP-hard optimization problem, which is difficult to solve using a classical convex optimization algorithm. From the above formula, it can be seen that the optimization objective is jointly determined by the total latency and energy consumption of the system. The communication resource allocation and computing resource allocation problems need to be solved jointly by allocating between DT models. In the computing resource management problem, the number of heterogeneous computing resources and DT models is also highly coupled, and the complex coupling between different communication states and optimization variables makes the optimization problem difficult to solve. In addition, the dynamic changes of UAVs terminal equipment and tasks have brought difficulties to the design of efficient resource management solutions in DT networks. Multi-agent deep reinforcement learning (MADRL) combines the perception ability of deep learning with the decision-making ability of reinforcement learning, which helps to learn near-optimal or near-equilibrium agent strategies in large-scale complex environments and realize collaborative decision-making of multiple decision-making entities. Therefore, this study models the joint objective optimization problem in the satellite-ground network as an MDP process, and proposes an algorithm based on DDPG to decide the allocation of bandwidth resources and the allocation ratio of computing resources, train the global model under the premise of satisfying constraints, and minimize the system latency and energy consumption.

[0049] Step 2.1: DDPG algorithm framework

[0050] DDPG algorithm is a DRL algorithm for continuous action space, which draws on the idea of ​​experience replay and target network of deep Q network. In the process of experience replay, the experience data samples obtained by the interaction between the agent and the environment are stored in the experience pool. When updating the network parameters, batch sample data is randomly extracted for training, which can break the correlation and dependency between data samples and improve the convergence speed of the algorithm. However, since the greedy strategy of Q-learning cannot be quickly implemented in the continuous action space, DDPG cannot directly adopt the Q-learning framework, but the actor-critic algorithm framework based on deterministic strategy.

[0051] 1. Critic network: The goal of the Critic network is to update the parameters by minimizing the error with the target Q value. The target Q value is calculated by the current reward and the Q value of the next state, and the formula is as follows:

[0052] y=r(s,a)+γQ'(s',μ'(s')) (5)

[0053] Where μ' is the target Actor network, Q'(s',μ'(s')) is the estimated Q value of the target Critic network for the next state s' and its target action μ'(s'), γ is the discount factor, and γ∈[0,1], r represents the reward value obtained by the agent after taking action a in state s. The Critic network updates the network parameters by minimizing the following loss function:

[0054]

[0055] Where N is the number of samples in a single batch. The Q function and the target Q are respectively expressed by parameters θ Q and θ Q' The simulation shows that the TargetCritic network is updated using the gradient descent method, which can be expressed as:

[0056] θ Q' =τθ Q +(1-τ)θ Q' (7)

[0057] Among them, the learning rate τ∈(0,1).

[0058] 2. Actor network: The goal of the Actor network is to update the parameters by maximizing the Q value of the Critic network. Therefore, the Actor network uses the gradient ascent method to update the parameters. The gradient of the Actor network parameters is calculated as follows:

[0059]

[0060] in, is the gradient of the Actor network parameters, is the gradient of the Q value of the action by the Critic network, is the policy gradient of the Actor network to the state. The Target Actor network is updated using the gradient ascent method and can be expressed as:

[0061] θ μ' =τθ μ +(1-τ)θ μ' (9)

[0062] Step 2.2: DDPG-based MADRL solution

[0063] In order to solve the problem of jointly optimizing latency and energy consumption in equation (4), a resource optimization solution based on DDPG was developed. The MDP model of this optimization problem can be expressed as (S, A, R, γ, p), where S is the state space of the environment, A is the action space, R is the reward function, γ is the discount factor of the reward function, and the state transition probability γ is the probability of the agent transferring from the current state to the next state after performing a certain action. The algorithm model is shown in the attached figure. Figure 2 shown.

[0064] The DDPG-based MADRL solution is based on the Actor-Critic algorithm framework. The Actor-Critic network architecture is shown in the attached figure. Figure 3 As shown. The architecture of the Actor network includes an input layer, a hidden layer, and an output layer. The input layer receives the state vector of the network environment, which is processed by the first fully connected layer and the ReLU activation function, and then input into the second fully connected layer, where the ReLU activation function is applied again for nonlinear transformation. Finally, the action vector is output through a fully connected layer, and the Tanh activation function is used to ensure that the action output is within a specific range, and the action is multiplied by the maximum action value accepted by the environment. The architecture of the Critic network is similar, but the input layer receives the concatenated vector of the state and action. After being processed by two fully connected layers and the ReLU activation function, a scalar Q value is output, which is used to evaluate the value of the state-action pair.

[0065] (1) State space: The state space S describes the physical operating status information of all UAVs and satellites within a specific range, as well as the amount of data for task k. For task k, the environmental state can be symbolically expressed as:

[0066] s k ={D1(k),...,D N (k),h1(k),...,h N (k),f1(k),...,f M (k)}(10)

[0067] Among them, D N (k),h N (k) represent the data size and channel gain of UAVs for task k respectively. M (k) is the computing resource of the satellite edge server.

[0068] (2) Action space: The action space contains all possible actions that the agent can take. For the scenario proposed in this invention, each UAV will be allocated a corresponding bandwidth resource and computing resource ratio, so the joint action of all agents can be expressed as:

[0069]

[0070] Among them, ξ k =(ξ k,1 ,ξ k,2 ,...ξ k,N ) represents the bandwidth allocation to UAVs, It represents the computing resource allocation ratio of satellite M to each UAV.

[0071] (3) Reward function: Reward is a function of state and action, which measures the effect of the action taken by the agent in a given state. The optimization problem proposed in this study contains two key performance indicators: latency and energy consumption. The optimization goal of the reinforcement learning problem is to maximize the long-term cumulative discounted reward. When the agent is in state s k When , execute action a k The immediate rewards are:

[0072]

[0073] In the optimization objective, the two elements of latency and energy consumption have different orders of magnitude. The element with a larger order of magnitude will dominate and have a greater impact on the result, thereby affecting the optimization of the other element. Therefore, this paper normalizes the latency and energy consumption and scales the reward features to avoid the impact of the order of magnitude differences between the reward elements on the model.

[0074] Step 2.3: Algorithm Pseudocode

[0075]

[0076] Table 1 System parameter description

[0077]

[0078]

Claims

1. A satellite-ground collaborative edge network resource allocation method based on deep deterministic policy gradient, characterized by: Design a satellite-ground collaborative edge network system that includes physical space and digital twin space. The satellite-ground collaborative edge network system describes the scenario of data unloading and data processing between UAVs and satellites, and models the computing and communication resource allocation problem in this scenario as a delay and energy consumption collaborative optimization problem with constraints. Use digital twin technology to map the operating status, association status, and network topology of UAVs and satellite entities into the edge network digital space, monitor the current operating status of the network, and assist in decision-making; Satellites, as aggregators and edge servers, perform verification and aggregation based on blockchain technology, and establish the allocation problem of communication resources and computing resources to balance the latency and energy consumption of the satellite-ground collaborative edge network system; secondly, in order to solve the NP-hard problem, the joint objective optimization problem in the satellite-ground collaborative edge network system is modeled as a Markov decision process MDP, and an algorithm based on DDPG is proposed to intelligently decide the allocation of network resources. Satellites, as intelligent agents, intelligently select actions according to the policy network. The state is defined as the amount of data generated by UAVs and the channel gain and the computing resource information of the satellite. The action is defined as the allocation of the number of channels and the allocation ratio of computing resources. The reward setting takes into account the latency and energy consumption factors of the satellite-ground collaborative edge network system; the DDPG algorithm gradually learns and optimizes the resource allocation decisions between UAVs and satellites by soft-updating the target network parameters, thereby improving the overall efficiency of the satellite-ground collaborative edge network system and reducing latency and energy consumption.

2. The satellite-ground collaborative edge network resource allocation method based on deep deterministic policy gradient according to claim 1 is characterized by: The method comprises the following steps, Step 1: Establish a digital twin-assisted satellite-ground collaborative edge network system; Step 1.1: Design of physical space and digital twin space; The proposed satellite-ground collaborative edge network system includes physical entity space and digital twin space; 1) Physical entity space; In the physical space, adjacent satellites in different orbits will cover a specific area on the ground at the same time. The UAVs in this area are grouped and associated with different satellites. Data is uploaded, models are sent, and real-time UAVs status information and network topology are updated according to the association relationship. In addition, the satellites use blockchain technology to propagate and aggregate models, and send the aggregated global model to UAVs. The satellite-ground collaborative edge network system assumes that a collection of N identical UAVs devices is included. The set of UAVs grouped and associated with M different edge servers is Using binary variables represents the relationship between UAVi and satellite j. If is 1, indicating that UAVi's data is unloaded to satellite j. 0, indicating that UAVi has no association with satellite j, that is, UAVi’s data is not unloaded to satellite j; the satellite processes the data unloaded by UAVs to train the local model, and at the same time, the satellite is regarded as a blockchain node to propagate and aggregate the model, and sends the aggregated global model to its associated UAVs, thereby achieving secure and reliable data transmission; 2) Digital twin space; In the digital twin space, the DT technology is used to map the operating status of UAVs and satellite entities to the edge network digital space, and the operating status is used to simulate the real UAVs and satellites. The real-time association status and network topology of the satellite as an edge server and UAVs are also mapped to the digital space of the edge server. Two DT models are designed, namely the DT model of UAVs and the DT model of satellites. The DT model of UAVs and the DT model of satellites are built into the satellite digital space to ensure the real-time status information mapping of UAVs equipment and satellites, and monitor the current operating status of the network. (1) DT model of UAVs; The DT of UAVs is a digital representation of UAVs, which continuously interacts with the entity and is updated in real time. For a specific task, the DT model of UAVi DT i The representation is as follows: Among them, f i represents the computing power of UAVi, D i represents the size of the UAVi preprocessed dataset, ξ i represents the number of channels allocated by UAVi, p i represents the transmit power of UAVi; (2) DT model of satellite; For satellite j, it is expressed as: Among them, f j represents the computing power of satellite j, DT i,j N represents the set of DT models mapped to UAVi in satellite j, DT,j represents the number of DT models of UAVs mapped to satellite j; Step 1.2: Communication model; The satellite-ground collaborative edge network system includes two different types of data transmission links: satellite-ground user link and inter-satellite high-speed link. The satellite-ground user link uses RF communication, and the inter-satellite link uses laser communication technology. In the satellite-to-ground user link, UAVs use multiplexing technology to transmit data to the satellite. The number of channels allocated by UAVi is ξ i ,ξ i It is one of the variables that affect the performance of the satellite-ground collaborative edge network system, which is determined by the edge node and satisfies 0<ξ i ≤ξ total ,ξ total is the total number of channels; the transmission rate of UAVi is as follows: Among them, B up represents the transmission bandwidth of a single uplink channel, N0 represents the noise power, and h i and p i denote the channel gain and transmit power of UAVi respectively; Step 1.3: Delay model; For task k, the total delay T of the satellite-ground collaborative edge network system to complete global model training total Including communication delay T trans and calculation delay T com Considering the delay requirement of each UAV task, the total delay from UAV uploading data to receiving the global model should be within T max within the scope; Communication delay T trans Including uplink data transmission delay Model propagation delay T of intersatellite link inter And the model sending delay T of the downlink down , The data transmission delay of UAVi in the uplink; in the uplink data transmission process, the bandwidth allocation of UAVs directly affects the performance and resource utilization efficiency of the satellite-ground collaborative edge network system; through the reasonable allocation of channels, the network load balancing is optimized to avoid excessive resource occupation by individual devices, thereby causing communication obstruction of other devices; Calculate the delay T com The processing delay T of the dataset containing the satellites associated with the UAVs train and the processing delay T of the block block ;The satellite needs to allocate appropriate computing resources to different UAVs’ data sets and dynamically adjust the usage ratio of computing resources according to the load conditions; Step 1.4: Energy consumption model; In the proposed satellite-ground collaborative edge network system, transmission energy consumption and computing energy consumption are two key indicators for evaluating network performance. The total energy consumption E total Energy consumption of transmitting the dataset containing all UAVs to the satellite Hovering energy consumption of all UAVs during the whole process And the computational energy consumption of training local models on all satellites in, and They represent the uplink data transmission energy consumption of UAVi and the hovering energy consumption of the entire process of UAVi, respectively. represents the computational energy consumption of satellite j for training the local model. The total energy consumption cannot exceed E max ; After the UAVs upload data to the satellite, the satellite allocates appropriate computing resources to the tasks of different UAVs and dynamically adjusts the usage ratio of computing resources according to the load conditions; Step 1.5: Joint optimization problem model; Under the condition of limited communication resources and computing resources, the delay and energy consumption can be minimized by reasonably allocating channel resources and computing resources during the training of the global model. The resource optimization goal in the satellite-ground collaborative edge network system is defined as: Among them, variables ξ and ρ are decision variables, representing the allocation of the number of channels and the allocation ratio of satellite computing resources respectively; constraint C1 represents the constraint on the weight coefficient, μ T and μ E are the weight coefficients of latency and energy consumption of the satellite-ground collaborative edge network system respectively; C2 means that for any satellite, the sum of computing resources allocated to its associated UAVs is less than its own total computing resources. represents the ratio of computing resources allocated to UAVi by satellite j; C3 indicates that the total number of DTs in all satellites must be equal to the number of UAVs; C4 ensures that the data of any UAV is uploaded to only one satellite, and C5 limits the allocation of bandwidth resources, indicating that the total number of channels allocated to UAVs must be less than the total number of channels ξ total ; C6 and C7 ensure the delay T of the satellite-ground collaborative edge network system total and the energy consumption E of UAV UAV To be better than the set threshold T th and E th , E UAV Energy consumption of transmitting the dataset containing UAVi to the satellite And the hovering energy consumption of UAVi throughout the process 3. The satellite-ground collaborative edge network resource allocation method based on deep deterministic policy gradient according to claim 2 is characterized by: Step 2: Design of resource management scheme based on DDPG algorithm; The above joint optimization problem model is an NP-hard optimization problem. The communication resource allocation and computing resource allocation problems need to be solved jointly by allocating them between DT models. In the computing resource management problem, the number of heterogeneous computing resources and DT models is highly coupled. The complex coupling between different communication states and optimization variables makes the optimization problem difficult to solve. The joint objective optimization problem in the satellite-ground collaborative edge network system is modeled as an MDP process, and a DDPG-based algorithm is proposed to decide the allocation of the number of channels and the allocation ratio of computing resources. The global model is trained under the premise of satisfying the constraints, while minimizing the latency and energy consumption of the satellite-ground collaborative edge network system. Step 2.1: DDPG algorithm framework; The DDPG algorithm framework is implemented based on the actor-critic algorithm framework of the deterministic strategy. The actor-critic algorithm framework includes: 1) Critic network: The goal of the critic network is to update the parameters by minimizing the error with the target Q value; the target Q value is calculated by the current reward r and the Q value of the next state, as follows: y=r(s,a)+γQ'(s',μ'(s')|θ Q' ) (5) Among them, r(s,a) represents the reward value obtained by the agent after taking action a in state s, and the Q function and target Q are respectively expressed by parameters θ Q and θ Q' Simulation representation, μ' is the target Actor network, Q'(s',μ'(s')|θ Q' ) is the estimated Q value of the target Critic network for the next state s' and its target action μ'(s'), γ is the discount factor, and γ∈[0,1]; the Critic network updates the network parameters by minimizing the following loss function: in, Indicates that the experience replay pool The expectation of the sampled (s,a,r,s') quadruple, Q(s,a|θ Q ) represents the Q value predicted by the Critic network based on the current state s and action a; the Target Critic network is updated using the gradient descent method, which is expressed as: i Q' =tθ Q +(1-τ)θ Q' (7) Among them, the soft update rate of the target network τ∈(0,1]; 2) Actor network: The goal of the Actor network is to update the parameters by maximizing the Q value of the Critic network. Therefore, the Actor network uses the gradient ascent method to update the parameters. The gradient of the Actor network parameters is calculated as follows: in, is the Actor network parameter θ μ The objective function J(θ μ ), represents the expected average operation on state s, which is obtained from the experience replay pool The sampling is obtained, is the gradient of the Q value of the Critic network for action a, a = μ(s|θ μ ) is the action a output by the Actor network according to the state s, Q(s,a|θ Q ) is the Q-value function estimated by the Critic network, is the Actor network strategy μ(s|θ μ ) for its parameter θ μ The TargetActor network is updated using the gradient ascent method, expressed as: i μ' =tθ μ +(1-τ)θ μ' (9) Among them, θ μ' represents the target Actor network parameters, the soft update rate of the target network τ∈(0,1]; Step 2.2: MADRL solution based on DDPG; In order to solve the problem of jointly optimizing delay and energy consumption in formula (4), a resource optimization problem based on DDPG is proposed; the MDP model of this optimization problem is expressed as (S, A, R, γ, p), where S is the state space of the environment, A is the action space, R is the reward function, γ is the discount factor of the reward function, and the state transition probability p is the probability of the agent transferring from the current state to the next state after performing a certain action; The DDPG-based MADRL solution is implemented based on the Actor-Critic network framework. In the Actor-Critic network architecture, the architecture of the Actor network includes an input layer, a hidden layer, and an output layer; the input layer receives the state vector of the network environment, and after being processed by the first fully connected layer and the ReLU activation function, it is input to the second fully connected layer, and the ReLU activation function is applied again for nonlinear transformation; finally, the action vector is output through a fully connected layer, and the Tanh activation function is used to ensure that the action output is within a specific range, and the action is multiplied by the maximum action value accepted by the environment; the input layer of the Critic network architecture receives the concatenated vector of the state and action, and after being processed by two fully connected layers and the ReLU activation function, a scalar Q value is output to evaluate the value of the state-action pair; (1) State space: The state space S describes the physical operating status information of all UAVs and satellites within a specific range, as well as the amount of data for task k. For task k, the environmental state can be symbolically expressed as: S k ={D1(k),...,D i (k),...D N (k),h1(k),...,h i (k),...h N (k),f1(k),...,f j (k),...f M (k)} (10) Among them, D i (k),h i (k) represents the data size and channel gain of UAVi for task k; f j (k) is the computing resources of satellite j; (2) Action space: The action space contains all possible actions taken by the agent. For the satellite-ground collaborative edge network system proposed in the present invention, each UAV will be allocated corresponding bandwidth resources and computing resources. Therefore, the joint action of all agents is expressed as: Among them, ξ k =(ξ k,1 ,ξ k,i ,...ξ k,N ) represents the number of channels allocated to UAVs, ξ k,i is the number of channels allocated to UAVi, represents the ratio of computing resources allocated by satellite j to each UAV, is the proportion of computing resources allocated by satellite j to UAVi; (3) Reward function: Reward is a function of state and action, which measures the effect of the action taken by the agent in a given state. The optimization problem contains two key performance indicators: latency and energy consumption. The optimization goal of the reinforcement learning problem is to maximize the long-term cumulative discounted reward. When the agent is in state s k When , execute action a k The immediate rewards are: In the optimization objective, the two elements of latency and energy consumption have different orders of magnitude. The element with a larger order of magnitude will dominate and have a greater impact on the result, thereby affecting the optimization of the other element. Therefore, the latency and energy consumption are normalized and the reward features are scaled to avoid the impact of the order of magnitude differences between the reward elements on the model.

Images