Satellite network resource management method based on deep reinforcement learning

By constructing a LEO satellite mobility and SIoT network model and combining it with deep reinforcement learning algorithms, the problem of limited computing and energy resources for IoT devices is solved, and efficient resource management of satellite-assisted Internet of Things systems is achieved.

CN119031394AActive Publication Date: 2024-11-26NANJING UNIV OF INFORMATION SCI & TECH

Patent Information

Application Number
CN202411521242.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-29
Publication Date
2024-11-26
Estimated Expiration
2044-10-29

AI Technical Summary

Technical Problem

In existing satellite-assisted Internet of Things (SIoT) systems, IoT devices have limited computing and energy resources, resulting in high latency and high energy consumption in sensor data processing, and a lack of effective resource management methods.

Method used

We construct an LEO satellite mobility model and an SIoT network model, combine deep reinforcement learning algorithms to optimize task offloading and resource allocation, achieve load balancing among terminals, edge computing and cloud computing nodes, and use the Model-Assisted Adaptive Deep Reinforcement Learning (MADRL) algorithm to minimize system overhead.

Benefits of technology

It effectively reduces system overhead, improves task processing efficiency, reduces latency and energy consumption, and achieves optimized resource allocation in various environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031394A_ABST
    Figure CN119031394A_ABST
Patent Text Reader

Abstract

The invention provides a satellite network resource management method based on deep reinforcement learning, and the method comprises the steps: 1, constructing an LEO satellite mobility model; step 2, constructing a SIoT network model with inter-satellite cooperation ISC enhancement; step 3, respectively calculating end-to-end time delay and system energy consumption under the three models of local calculation, edge calculation and cloud calculation; and step 4, establishing a task unloading and resource allocation algorithm and a task unloading decision based on deep reinforcement learning, and realizing load balancing among the terminal, the edge and the cloud computing node. Aiming at a dynamic task arrival scene, the invention provides a model-assisted adaptive deep reinforcement learning algorithm, which can realize joint configuration of task unloading decision, communication resources and computing resources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of information and communication technology, and in particular relates to a satellite network resource management method based on deep reinforcement learning. Background Art

[0002] The existing wide-area IoT network is mainly built on the basis of the 5th Generation Mobile Communication Technology (5G) ground network. Compared with the ground network, the satellite network is rapidly becoming an important communication infrastructure with its advantages of seamless coverage, flexibility and reliability, which is promoting the development of global interconnection and communication technology. Therefore, the satellite network can serve as a supplement to the ground network to provide services for IoT devices in remote suburbs or disaster areas, which is called satellite-assisted Internet of Things (SIoT). Since IoT devices usually have limited power and few available resources for communication, computing and caching, the sensor data generated by IoT devices usually need to be forwarded to the cloud or edge computing nodes for further processing. Mobile edge computing gives computing power to the edge of the network, and can effectively reduce processing latency by caching content and deploying edge computing servers near IoT devices. With the development of on-board processing technology, satellites with on-board processing units can also be regarded as edge computing nodes to provide computing services to ground users. In the current SIoT, the terminal forwards data to the ground cloud data center via satellite for processing. Compared with satellite nodes, the cloud has higher computing power and sufficient energy supply. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a satellite network resource management method based on deep reinforcement learning in view of the shortcomings of the prior art, including the following steps: Step 1, constructing a LEO satellite mobility model; Step 2, build a SIoT network model with inter-satellite collaboration ISC enhancement; Step 3: Calculate the end-to-end delay and system energy consumption under the three models of local computing, edge computing, and cloud computing respectively; Step 4: Establish a task offloading and resource allocation algorithm and task offloading decision based on deep reinforcement learning to achieve load balancing among terminals, edge and cloud computing nodes.

[0004] In step 1, the LEO satellite mobility model includes: LEO satellites at altitude above the ground H On track at speed Fly at a constant speed, For LEO satellites s The angle with the positive horizontal direction at time slot n, For LEO satellites s In time slot n The geometric angle corresponding to the remaining coverage arc length to satellite user m at time is, R is the radius of the Earth, when When , LEO satellite s can establish a communication link with device m; According to geometric relationship, when hour, It is expressed as: (1), when hour, It is expressed as: (2), The linear distance between satellite s and satellite user m in time slot n It is expressed as: (3).

[0005] Step 2 includes: establishing a system for analysis according to the SIoT network model, wherein the system includes M IoT devices, S LEO satellites and 1 cloud computing center, M=D+O, where O IoT devices are in the suburbs, D IoT devices are in the disaster area, and M IoT devices are a collection of , a collection of IoT devices in the disaster area , a collection of IoT devices in the suburbs , LEO satellites are represented as a set , To access the satellite, For cooperative satellites, the collection of time slots , N represents the total number of time slots.

[0006] In step 2, let the length of a time slot be , a cycle is divided into N time slots, and assume that the channel state remains unchanged within a time slot; n Mission Modeled as , Indicates the task Include bit data, need to be in time The workload is , the number of CPU cycles required to process the task ,use Indicates the task uninstall decision.

[0007] In step 2, a satellite channel model is established, including communication loss, rain attenuation and cloud attenuation, where communication loss It is expressed as: (4), in is the communication distance, is the wavelength, is the carrier frequency, c is the speed of light; Rain attenuation for: (5), in, For rainfall rates exceeding 0.01% per year, is a valid path, and b are regression coefficients related to raindrop size distribution, temperature, and frequency; Cloud attenuation for: (6), in L is the total columnar content of liquid water in the cloud, is the specific attenuation coefficient of the cloud layer, and are the real and imaginary parts of the dielectric constant of water, respectively; The total channel fading h during satellite communication is expressed as: (7).

[0008] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0009] The present invention proposes a dynamic SIoT network based on terminal-edge-cloud, and uses ISC (Inter-Satellite Collaboration) technology to improve the performance of the edge. In the constructed network, the tasks generated by the IoT device can be processed locally, offloaded to LEO (Low Earth Orbit) for processing, or forwarded to the cloud computing center for processing through the LEO satellite, that is, the tasks can be offloaded to the most suitable node to achieve load balancing between the terminal, edge and cloud computing nodes. In addition, the present invention considers the requirements of delay and energy consumption at the same time, takes the weighted sum of the delay and energy consumption of task processing as the overhead, and determines the weight between delay and energy consumption according to different task processing methods. The task offloading decision, CPU operating frequency and transmission power joint allocation problem of the SIoT network are modeled as a mixed integer nonlinear programming (Mixed Integer Nonlinear Programming, MINLP) problem with resource constraints. In order to analyze and solve this problem, the present invention combines traditional optimization algorithms with deep reinforcement learning algorithms, and proposes a model-assisted adaptive deep reinforcement learning (MADRL) algorithm to minimize system overhead.

[0010] The present invention has the following beneficial effects: The present invention effectively realizes the optimization of task offloading decision and resource allocation by introducing a model-assisted adaptive deep reinforcement learning (MADRL) algorithm. By introducing the mobility of satellites and ISC technology, the system overhead is significantly reduced. In addition, compared with traditional algorithms, the algorithms included in the present invention show lower system overhead in various test environments, indicating that they have significant advantages in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is the geometric relationship diagram between LEO satellite s and satellite user m.

[0012] Figure 2 It is a schematic diagram of the system model of the satellite in the first half of a cycle.

[0013] Figure 3 It is a schematic diagram of the system model of the satellite in the second half of a cycle.

[0014] Figure 4 It is the flow chart of MADRL algorithm.

[0015] Figure 5 The change of algorithm reward value with the number of iterations under different learning rates.

[0016] Figure 6 The cumulative cost of the algorithm changes over time under different learning rates.

[0017] Figure 7 The change of algorithm reward value with the number of iterations under different attenuation factors.

[0018] Figure 8 The cumulative overhead of the algorithm proposed in the present invention varies with time under different exploration attenuation factors. DETAILED DESCRIPTION

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0020] This embodiment provides a satellite network resource management method based on deep reinforcement learning, including: Step 1, constructing a LEO satellite mobility model; Step 2, build a SIoT network model with inter-satellite collaboration ISC enhancement; Step 3: Calculate the end-to-end delay and system energy consumption under the three models of local computing, edge computing, and cloud computing respectively; Step 4: Establish a task offloading and resource allocation algorithm and task offloading decision based on deep reinforcement learning to achieve load balancing among terminals, edge and cloud computing nodes.

[0021] In step 1, the LEO satellite mobility model includes: LEO Satellites s The geometric relationship between the satellite user m is as follows: Figure 1 Considering the mobility of LEO satellites, LEO satellites are at a certain height above the ground. H On track at speed Fly at a constant speed, For LEO satellites s The angle with the positive horizontal direction at time slot n, For LEO satellites s In time slot n The geometric angle corresponding to the remaining coverage arc length to satellite user m at time is, R is the radius of the Earth, when When , LEO satellite s can establish a communication link with device m; According to geometric relationship, when hour, It is expressed as: (1), when hour, It is expressed as: (2), The linear distance between satellite s and satellite user m in time slot n It is expressed as: (3).

[0022] Step 2 includes: Figure 2 As shown in the figure, a system for analysis is established based on the SIoT network model. The system includes M IoT devices, S LEO satellites and 1 cloud computing center, M=D+O, where O IoT devices are in the suburbs, D IoT devices are in the disaster area, and M IoT devices are a collection of , a collection of IoT devices in the disaster area , a collection of IoT devices in the suburbs , LEO satellites are represented as a set , To access the satellite, For cooperative satellites, the collection of time slots , N represents the total number of time slots.

[0023] In step 2, let the length of a time slot be , a cycle is divided into N time slots, and assume that the channel state remains unchanged within a time slot; n Mission Modeled as , Indicates the task Include bit data, need to be in time The workload is , the number of CPU cycles required to process the task ,use Indicates the task uninstall decision.

[0024] In step 2, for satellite communication, channel fading is completely different from that of terrestrial communication. Here, this embodiment considers a satellite channel model that is closer to the real scene, including communication loss, rain attenuation and cloud attenuation, where communication loss It is expressed as: (4), in is the communication distance (in km), is the wavelength, is the carrier frequency (in GHz), c is the speed of light; When the carrier frequency is above 10 GHz, rainfall is one of the main causes of satellite communication signal attenuation. for: (5), in, For rainfall rates exceeding 0.01% per year, is a valid path, and b are regression coefficients related to raindrop size distribution, temperature, and frequency; Cloud attenuation for: (6), in L is the total columnar content of liquid water in the cloud, is the specific attenuation coefficient of the cloud layer, and are the real and imaginary parts of the dielectric constant of water, respectively; The total channel fading h during satellite communication is expressed as: (7).

[0025] Step 3 includes: Step 3-1, build a local computing model: Assume that satellite user m is in time slot n The CPU operating frequency is , satellite user m in time slot n Task processing delay It is expressed as: (8), Satellite user m in time slot n Task processing energy consumption It is expressed as: (9), in Indicates the electrical coefficient; Satellite user m in time slot n Expenses It is expressed as: (10), in, is the delay sensitivity coefficient in local computing mode; Step 3-2, build edge computing model: Suppose the moment of the intermediate state of the satellite motion is ,like Figure 2 As shown, when At that time, the disaster area and the cloud computing center were connected by satellite. Within the service area of ​​satellite Within the service range, t represents the moment of the satellite motion state; Figure 3 As shown, when When the cloud computing center is on the satellite Within the service area of ​​the satellite within the service scope; when inter-satellite collaborative computing tasks are required, the task transmission link involves ISL (Inter-Satellite Link).

[0026] Satellite , In time slot n The CPU operating frequencies are and The transmission powers are and ; The signal transmission rate is calculated by Shannon's formula. It is assumed that IoT devices have a sufficient number of orthogonal channels and the channel allocation between multiple devices can be ignored. This assumption is based on the application of multi-frequency time division multiple access technology, which can dynamically allocate these orthogonal channels according to current needs and traffic conditions. This enables the system to effectively manage available resources and ensure that a sufficient number of channels are provided for devices for data transmission.

[0027] Satellite user m in time slot n Delay It is expressed as: (11), in, It means that the mission transmission involves inter-satellite cooperation and requires inter-satellite links; It means that the mission transmission link does not involve inter-satellite cooperation and does not require inter-satellite links; Indicates the first half of a satellite cycle. Indicates that the satellite is in the second half of a cycle; Indicates that satellite user m offloads the task to the satellite The signal transmission rate, Indicates that satellite user m offloads the task to the satellite The signal transmission rate, Indicates the signal transmission rate between the access satellite and the cooperative satellite. Denotes the satellite user m and the satellite The distance between Denotes the satellite user m and the satellite The distance between Indicates the distance between the access satellite and the cooperative satellite; Energy consumption of satellite user m in time slot n It is expressed as: (12), in, Indicates that satellite user m is in time slotn The transmission power of In edge computing mode, satellite user m is in time slot n Expenses for: (13), in, is the delay sensitivity coefficient in edge computing mode; Step 3-3, build a cloud computing model: Assume that the single-core CPU operating frequency of the cloud computing center is , the number of cores is ; The delay of satellite user m in time slot n is It is expressed as: (14), in Indicates satellite Offloading tasks to satellites The signal transmission rate, Indicates satellite The signal transmission rate of offloading tasks to the cloud computing center, Indicates satellite With satellite The distance between Indicates satellite Distance to the cloud computing center; Energy consumption of satellite user m in time slot n It is expressed as: (15), Satellite user m in time slot under cloud computing mode n Total expenses for: (16), in, is the delay sensitivity coefficient in the cloud computing model; Step 3-4, problem optimization: Satellite user m in time slot n End-to-end latency of tasks generated under different processing methods and energy consumption They are: (17), (18), in Indicates the task processing method. When it is equal to 0, it means that the task is processed locally. When it is equal to 1, it means that the task is offloaded to the access satellite for processing. When it is equal to 2, 3, 4, or 5, it means that the task is offloaded to the cooperative satellite for processing. When it is equal to 6, it means that the task is offloaded to the cloud computing center for processing; a simple analysis shows that local processing consumes less energy, but due to limited computing power, it may not be able to meet the latency requirements of task processing. Processing tasks at edge nodes is beneficial to reducing processing latency, but increases satellite energy consumption. Processing tasks at the cloud center reduces computing latency, but increases propagation latency. Therefore, the offloading decision and resource allocation of each time slot have a significant impact on the task completion latency and energy consumption. The present invention studies the problem of how to minimize the weighted sum of system latency and energy consumption under latency and energy consumption constraints within a cycle in a dynamic network scenario. Task offloading decisions, communication, and computing resource allocation are jointly optimized. Formula (19) gives the target optimization problem P1.

[0028] The optimization goal is to minimize the weighted sum of system delay and energy consumption in one cycle. The optimization variables are task offloading decision, transmission power and CPU operating frequency of IoT devices, and transmission power and CPU operating frequency of LEO satellites. The delay sensitivity coefficient is different under different processing methods. Because at the local node, the device needs to ensure battery life and is sensitive to energy consumption, the delay sensitivity coefficient should be set smaller. At the edge node, low-orbit satellites can reduce processing delays, but increase communication delays. At the same time, the computing power of low-orbit satellites is limited, and it is very sensitive to delays and energy consumption. In cloud nodes, the propagation delay increases significantly, and the computing power of the cloud center is much higher than that of devices and LEO satellites. Therefore, it is sensitive to delays, and the delay sensitivity coefficient should be set relatively high.

[0029] (19), in, Indicates IoT devices m In time slot n Generate the maximum tolerable delay of the task, Indicates IoT devices m The maximum energy consumption, Indicates satellite s The maximum energy consumption, Indicates IoT devices m The maximum CPU operating frequency, Indicates IoT devices m The maximum transmission power, Indicates satellite s The maximum CPU operating frequency, Indicates satellite s The maximum transmission power; Constraints C9 and C10 are continuity constraints, which ensure that the state changes of the system at adjacent moments will not be too drastic. Represents a non-zero positive number, which is used to measure the closeness of the function value at the next moment to the current moment. To constrain C9 The value of To constrain C10 The value of represents the delay sensitivity coefficient, that is , and One of them; Constraint C1 indicates that the task processing methods include local computing, offloading to LEO satellites (including access satellites and cooperative satellites) and offloading to cloud computing centers. Constraint C2 indicates that the end-to-end delay of task processing in any time slot must be less than the delay required by the task. Constraint C3 indicates that the energy consumption of satellite user m in any time slot must be less than its maximum energy consumption. Constraint C4 indicates that the satellite s The energy consumption of satellite user m must be less than its maximum energy consumption. Constraint C5 indicates that the CPU operating frequency of satellite user m must be less than its maximum operating frequency at any time slot. Constraint C6 indicates that the transmission power of satellite user m must be less than its maximum power at any time slot. Constraint C7 indicates that the transmission power of satellite user m must be less than its maximum power at any time slot. s The CPU operating frequency must be less than its maximum operating frequency. Constraint C8 indicates that in any time slot, the satellite s The transmission power of the system must be less than its maximum power. Constraints C9 and C10 are continuity constraints, which ensure that the state changes of the system at adjacent moments will not be too drastic.

[0030] Step 4 includes: In formula (19), the CPU operating frequency and transmission power are continuous variables, and the unloading decision is a discrete variable. The objective function is nonlinear with respect to these variables. Therefore, formula (19) is a MINLP problem. Traditional optimization methods are difficult to find the optimal solution. The present invention uses a MADRL algorithm to solve this problem. The first layer optimizes the CPU operating frequency and transmission power with the assistance of a model and a binary search algorithm. The second layer uses an adaptive DRL algorithm to adapt to dynamic network scenarios and generates a Q network through self-learning. The task unloading and resource allocation algorithm based on deep reinforcement learning includes: optimizing the CPU operating frequency and transmission power with the assistance of a model: When the task is computed locally, the optimization problem is transformed into: (20), The cost function F1 in local computing mode is expressed as (twenty one), Extreme Points for , constraints C1 and C2 are simplified to , ; Upper bound of the feasible solution range of F1 and the lower bound for: (twenty two), The optimal cost of local computation is expressed as: (twenty three), in Indicates that the cost function takes an upper bound The value of Indicates that the cost function is at the extreme point The value of Denotes the cost function to be lower bounded The value at time; When edge computing is used for the task, the optimization problem is transformed into: (twenty four), When the task adopts cloud computing, the optimization problem is converted to: (25).

[0031] In step 4, the task offloading decision is determined using a Deep Q-Network (DQN) and a Double Deep Q-Network (DDQN). There are two neural networks in DQN, namely the online network and the target network. The online network is used to select action a. When the state state is input, the Q value of each action is output, and then the action a corresponding to the maximum Q value is selected. The target network is used to estimate the Q value to update the model parameters. The parameter update method is to cover the target network with the online network every X steps. Since the Q value output by the target network in DQN will produce an overestimation, it is essentially because the action selected by the target network is not correct, resulting in an inappropriate estimated Q value. DDQN solves this problem. The action selection in DDQN is directly done by the online network. Therefore, the overall structure of DDQN is roughly the same as DQN. The only difference is that the calculation method of y value is different during the learning process. In DDQN, the calculation of the target value y is related to both the target network and the online network. That is, the state is first input into the online network to determine the action a. Then the state is input into the target network to output the Q value of all actions, and the Q value of action a is directly used to calculate the target value y. Even if the Q value corresponding to action a in the output of the target network is not the largest, it is still used to calculate the target value y. However, when the amount of data is small, DDQN is usually not as good as DQN in performance. This is because when the amount of data is small, the parameters of the two networks are difficult to train effectively, which makes the estimation of the target value have a large deviation, thereby reducing the performance of DDQN. This corresponds to the different situations in the disaster area and the suburbs. The disaster area has a high population density, a large number of devices, and a large amount of data generated, which is more suitable for DDQN processing. The suburbs have a low population density and a small amount of data, and the DQN algorithm can achieve better performance. Therefore, this embodiment proposes an adaptive DRL algorithm to train the Q networks of the suburbs and disaster areas, and then obtain task offloading decisions. The adaptive deep reinforcement learning DRL algorithm includes the following steps: Step a1, initialization and preparation: Step a1-1, initializing the parameters of the online network Q and the target network Q_hat; Step a1-2, set the training parameters, including learning rate lr, exploration rate epsilon, discount factor γ, target network update frequency target_update_freq, exploration rate decay epsilon_decay, size of experience replay pool D, minimum batch size batch_size and number of algorithm running rounds num_episode; Step a2, perform the following training process: Step a2-1, for each episode from 1 to num_episode, perform the following operations: Step a2-1-1, initializing state state; Step a2-1-2, for each step, perform the following operations: Step a2-1-1-1, randomly select an action a with probability epsilon; Step a2-1-1-2, otherwise, select action a that maximizes Q(state, a); if the current step does not select a random action with probability epsilon (that is, if a probability-based "if" condition check is passed and the condition is not met, that is, no random action is selected), then the steps after "otherwise" will be executed. Q(state, a) represents the expected return of performing action a in state state; Step a2-1-1-3, execute action a, observe the reward r and new state state'; Step a2-1-1-4, store the transfer tuple (state, a, r, state') into the experience replay pool D; Step a2-1-1-5, update the state state to state'; Step a2-2: If the size of the experience replay pool D is greater than or equal to batch_size, perform the following operations: Step a2-2-1, randomly extract a batch of transition tuples (state, a, r, state') from the experience replay pool D; Step a2-2-2, for each extracted transition tuple, calculate the target value y: if state' is the terminal state, then y=r, otherwise, for the deep Q network DQN, ; For the Dual Deep Q Network DDQN , ;in max_a' represents the maximum value among all possible actions a'; Q_hat(state', a') represents the expected return of executing action a' in state state'; argmax_a' represents the action a' that maximizes the Q value; Q(state', a') represents the expected return of executing action a' in state state'; Step a2-2-3, calculate the loss function using the calculated target value y and the expected return of a certain action predicted under a given state; Step a2-2-4, using the gradient descent method to update the parameters of the online network Q; Step a2-3, every target_update_freq steps, copy the parameters of the online network Q to the target network Q_hat; Step a2-4, after all rounds are completed, return the trained online network Q.

[0032] In step 4, in order to achieve a comprehensive and effective solution, the optimization strategy is comprehensively considered, the model-assisted resource allocation is combined with the learning optimization offloading decision, and the model-assisted adaptive deep reinforcement learning (MADRL) is proposed. The MADRL algorithm flow chart is as follows: Figure 4 As shown in the figure, the first layer optimizes the CPU operating frequency and transmission power through model assistance and binary search algorithm, solves the optimal cost of each task situation and processing method, and obtains the optimal cost matrix cost_matrix_disaster and cost_matrix_remote for the disaster area and remote suburbs. The second layer uses the adaptive DRL algorithm to train the Q network and obtain the task offloading decision according to the actual task status.

[0033] The various elements in the Q network are described as follows: The deep Q network DQN includes the following elements: State space: In each time slot, the system observes the current state and obtains environmental information; the state is represented by the task state of each user, which includes the task size , Workload and maximum tolerable delay , ; The state space at time slot n is defined as ,in Indicates the device M In time slot n The size of the task, Indicates the device M The task load in time slot n is Indicates the device M Generate the maximum tolerable delay of a task in time slot n; Action space: The online network gets the state space After that, the corresponding discrete unloading decision will be generated. , , ; Reward function: After taking action a in state state, the environment enters the next state state' and returns a reward r, where the reward value r is defined as the inverse of the system cost.

[0034] In a specific embodiment of the present invention, the performance of the proposed algorithm is evaluated through simulation analysis and the performance of the proposed algorithm is compared with that of a benchmark algorithm.

[0035] The main parameter settings used in the simulation include: for the number of devices, assume D=300, O=5. At the same time, the MADRL algorithm runs num_episode=1000 rounds, batch size batch_size=32, learning rate lr=0.001, discount factor γ=0.9, initial exploration probability epsilon=1, exploration probability decay rate epsilon_decay=0.995, and minimum exploration probability epsilon_min=0.01. Based on the laser link deployment, the communication capacity of ISL is set to 10Gbps. The number of devices in the disaster area is 300, the number of devices in the suburbs is 5, the task size is [1e2,1e3,1e4,1e5,1e6]bit, the task load is [1,1.5]kcycle / bit, the maximum tolerable delay is [0.05,0.1]s, the electrical coefficient is 10^(-28), and the channel bandwidth is 10Mb; The antenna gain is 20dB, the noise temperature is 290°, the maximum energy consumption of IoT satellite user m is 5w, the maximum energy consumption of LEO satellite s is 2000w, the single-core CPU operating frequency of the cloud computing center is 1.45GHz, the number of cores of the cloud computing center is 256, the number of episodes num_episode is 1000, the batch size batch_size is 32, the learning rate lr is 0.001, the discount factor γ is 0.9, the initial exploration probability epsilon is 1, the exploration probability decay rate epsilon_decay is 0.995, and the minimum exploration probability epsilon_min is 0.01.

[0036] Through a series of simulation experiments, the effects of learning rate and exploration rate on the convergence performance of the proposed algorithm are compared. Figure 5 The figure shows how the algorithm reward value changes with the number of iterations under different learning rates. When the learning rate is 0.1 and 0.01, the training effect is not good, and the reward value fluctuates violently with the increase in the number of iterations. When the learning rate is reduced to 0.001 and 0.0001, the convergence performance is better, and the reward value fluctuates little as the training progresses. Figure 6 The cumulative overhead of the algorithm changes over time at different learning rates. When the learning rate is 0.001, the unloading decision made by the trained model minimizes the system overhead, so the present invention selects a learning rate of 0.001. Figure 7 The change of algorithm reward value with the number of iterations under different attenuation factors. When the attenuation factor is too large, the exploration probability decreases too slowly, and a large amount of exploration is still performed in the later stage of training, which will cause the model to fail to fully utilize the learned knowledge and fail to converge to the optimal strategy. When the attenuation factor is too small, the exploration probability decreases too quickly, and the existing strategy is over-utilized in the early stage of training, which may cause the Q network to fail to fully explore the environment and fall into the local optimum. Therefore, the choice of the attenuation factor should not be too large or too small. Figure 8The cumulative overhead of the algorithm proposed in this invention changes with time under different exploration attenuation factors. When the attenuation factor is 0.95 and 0.9995, the system overhead is greater than 0.995. Therefore, this algorithm sets the attenuation factor to 0.995, which can find a good balance between exploration and utilization. It can fully explore in the early stage of training and better utilize the learned strategies in the later stage, so that the reward value can converge to a higher level quickly.

[0037] The present invention provides a satellite network resource management method based on deep reinforcement learning. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A satellite network resource management method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1, constructing a LEO satellite mobility model; Step 2, build a SIoT network model with inter-satellite collaboration ISC enhancement; Step 3: Calculate the end-to-end delay and system energy consumption under the three models of local computing, edge computing, and cloud computing respectively; Step 4: Establish a task offloading and resource allocation algorithm and task offloading decision based on deep reinforcement learning to achieve load balancing among terminals, edge and cloud computing nodes.

2. The method according to claim 1, characterized in that In step 1, the LEO satellite mobility model includes: LEO satellites at altitude above the ground H On track at speed Fly at a constant speed, For LEO satellites s The angle with the positive horizontal direction at time slot n, For LEO satellites s In time slot n The geometric angle corresponding to the remaining coverage arc length to satellite user m at time is, R is the radius of the Earth, when When , LEO satellite s can establish a communication link with device m; According to geometric relationship, when hour, It is expressed as: (1), when hour, It is expressed as: (2), The linear distance between satellite s and satellite user m in time slot n It is expressed as: (3)。 3. The method according to claim 2, characterized in that Step 2 includes: establishing a system for analysis according to the SIoT network model, wherein the system includes M IoT devices, S LEO satellites and 1 cloud computing center, M=D+O, where O IoT devices are in the suburbs, D IoT devices are in the disaster area, and M IoT devices are a collection of , a collection of IoT devices in the disaster area , a collection of IoT devices in the suburbs , LEO satellites are represented as a set , To access the satellite, For cooperative satellites, the collection of time slots , N represents the total number of time slots.

4. The method according to claim 3, characterized in that In step 2, let the length of a time slot be , assuming that the channel state remains unchanged within a time slot; n Mission Modeled as , Indicates the task Include bit data, need to be in time The workload is , the number of CPU cycles required to process the task ,use Indicates the task uninstall decision.

5. The method according to claim 4, characterized in that In step 2, a satellite channel model is established, including communication loss, rain attenuation and cloud attenuation, where communication loss It is expressed as: (4), in is the communication distance, is the wavelength, is the carrier frequency, c is the speed of light; Rain attenuation for: (5), in, For rainfall rates exceeding 0.01% per year, is a valid path, and b are regression coefficients related to raindrop size distribution, temperature, and frequency; Cloud attenuation for: (6), in L is the total columnar content of liquid water in the cloud, is the specific attenuation coefficient of the cloud layer, and are the real and imaginary parts of the dielectric constant of water, respectively; The total channel fading h during satellite communication is expressed as: (7)。 6. The method according to claim 5, characterized in that Step 3 includes: Step 3-1, build a local computing model: Assume that satellite user m is in time slot n The CPU operating frequency is , satellite user m in time slot n Task processing delay It is expressed as: (8), Satellite user m in time slot n Task processing energy consumption It is expressed as: (9), in Indicates the electrical coefficient; Satellite user m in time slot n Expenses It is expressed as: (10), in, is the delay sensitivity coefficient in local computing mode; Step 3-2, build edge computing model: Suppose the moment of the intermediate state of the satellite motion is ,when At that time, the disaster area and the cloud computing center were connected by satellite. Within the service area of ​​satellite Within the service range, t represents the moment of satellite motion; when When the cloud computing center is on the satellite Within the service area of ​​the satellite within the scope of services; Satellite , In time slot n The CPU operating frequencies are and The transmission powers are and ; Satellite user m in time slot n Delay It is expressed as: (11), in, It means that the mission transmission involves inter-satellite cooperation and requires inter-satellite links; It means that the mission transmission link does not involve inter-satellite cooperation and does not require inter-satellite links; Indicates the first half of a satellite cycle. Indicates that the satellite is in the second half of a cycle; Indicates that satellite user m offloads the task to the satellite The signal transmission rate, Indicates that satellite user m offloads the task to the satellite The signal transmission rate, Indicates the signal transmission rate between the access satellite and the cooperative satellite. Denotes the satellite user m and the satellite The distance between Denotes the satellite user m and the satellite The distance between Indicates the distance between the access satellite and the cooperative satellite; Energy consumption of satellite user m in time slot n It is expressed as: (12), in, Indicates that satellite user m is in time slot n The transmission power of In edge computing mode, satellite user m is in time slot n Expenses for: (13), in, is the delay sensitivity coefficient in edge computing mode; Step 3-3, build a cloud computing model: Assume that the single-core CPU operating frequency of the cloud computing center is , the number of cores is ; The delay of satellite user m in time slot n is It is expressed as: (14), in Indicates satellite Offloading tasks to satellites The signal transmission rate, Indicates satellite The signal transmission rate of offloading tasks to the cloud computing center, Indicates satellite With satellite The distance between Indicates satellite Distance to the cloud computing center; Energy consumption of satellite user m in time slot n It is expressed as: (15), Satellite user m in time slot in cloud computing mode n Total expenses for: (16), in, is the delay sensitivity coefficient in the cloud computing model; Step 3-4, problem optimization: Satellite user m in time slot n End-to-end latency of tasks generated under different processing methods and energy consumption They are: (17), (18), in Indicates the task processing mode. When it is equal to 0, it means that the task is processed locally. When it is equal to 1, it means that the task is offloaded to the access satellite for processing. When it is equal to 2, 3, 4, or 5, it means that the task is offloaded to the cooperative satellite for processing. When it is equal to 6, it means that the task is offloaded to the cloud computing center for processing; The target optimization problem P1 is: (19), in, Indicates IoT devices m In time slot n Generate the maximum tolerable delay of the task, Indicates IoT devices m The maximum energy consumption, Indicates satellite s The maximum energy consumption, Indicates IoT devices m The maximum CPU operating frequency, Indicates IoT devices m The maximum transmission power, Indicates satellite s The maximum CPU operating frequency, Indicates satellite s Maximum transmission power; Represents a non-zero positive number, which is used to measure the closeness of the function value at the next moment to the current moment. To constrain C9 The value of To constrain C10 The value of Indicates the delay sensitivity coefficient.

7. The method according to claim 6, characterized in that In step 4, the task offloading and resource allocation algorithm based on deep reinforcement learning includes: optimizing the CPU operating frequency and transmission power through model assistance: When the task is computed locally, the optimization problem is transformed into: (20), The cost function F1 in local computing mode is expressed as (21), Extreme Points for , constraints C1 and C2 are simplified to , ; Upper bound of the feasible solution range of F1 and lower bound for: (22), The optimal cost of local computation is expressed as: (23), in Indicates that the cost function takes an upper bound The value of Indicates that the cost function is at the extreme point The value of Denotes the cost function to be lower bounded The value at time; When edge computing is used for the task, the optimization problem is transformed into: (24), When the task adopts cloud computing, the optimization problem is converted to: (25)。 8. The method according to claim 7, characterized in that In step 4, the task offloading decision is determined using a deep Q network DQN and a dual deep Q network DDQN, wherein the deep Q network DQN includes an online network Q and a target network Q_hat; An adaptive deep reinforcement learning (DRL) algorithm is used to train the deep Q network (DQN) in the suburbs and disaster areas, and then obtain the task offloading decision. The adaptive deep reinforcement learning (DRL) algorithm includes the following steps: Step a1, initialization and preparation: Step a1-1, initializing the parameters of the online network Q and the target network Q_hat; Step a1-2, set the training parameters, including learning rate lr, exploration rate epsilon, discount factor γ, target network update frequency target_update_freq, exploration rate decay epsilon_decay, size of experience replay pool D, minimum batch size batch_size and number of algorithm running rounds num_episode; Step a2, perform the following training process: Step a2-1, for each episode from 1 to num_episode, perform the following operations: Step a2-1-1, initializing state state; Step a2-1-2, for each step, perform the following operations: Step a2-1-1-1, randomly select an action a with probability epsilon; Step a2-1-1-2, otherwise, select action a that maximizes Q(state, a); Q(state, a) represents the expected return of executing action a in state state; Step a2-1-1-3, execute action a, observe the reward r and new state state'; Step a2-1-1-4, store the transfer tuple (state, a, r, state') into the experience replay pool D; Step a2-1-1-5, update the state state to state'; Step a2-2: If the size of the experience replay pool D is greater than or equal to batch_size, perform the following operations: Step a2-2-1, randomly extract a batch of transition tuples (state, a, r, state') from the experience replay pool D; Step a2-2-2, for each extracted transition tuple, calculate the target value y: if state' is the terminal state, then y=r, otherwise, for the deep Q network DQN, ; For the Dual Deep Q Network DDQN , ;in max_a' represents the maximum value among all possible actions a'; Q_hat(state', a') represents the expected return of executing action a' in state state'; argmax_a' represents the action a' that maximizes the Q value; Q(state', a') represents the expected return of executing action a' in state state'; Step a2-2-3, calculate the loss function using the calculated target value y and the expected return of a certain action predicted under a given state; Step a2-2-4, using the gradient descent method to update the parameters of the online network Q; Step a2-3, every target_update_freq steps, copy the parameters of the online network Q to the target network Q_hat; Step a2-4, after all rounds are completed, return the trained online network Q.

9. The method according to claim 8, characterized in that In step 4, the deep Q network DQN includes the following elements: State space: In each time slot, the system observes the current state and obtains environmental information; the state is represented by the task state of each user, which includes the task size , Workload and maximum tolerable delay , ; The state space at time slot n is defined as ,in Indicates the device M In time slot n The size of the task, Indicates the device M The task load in time slot n is Indicates the device M Generate the maximum tolerable delay of a task in time slot n; Action space: The online network gets the state space After that, the corresponding discrete unloading decision will be generated. , , ; Reward function: After taking action a in state state, the environment enters the next state and returns a reward r, where the reward value r is defined as the inverse of the system overhead.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Calculation unloading method for distributed deep learning in satellite-ground collaborative network

    CN114153572A

  • Unloading method and device for satellite cloud edge cooperative computing

    CN117714446A

  • Satellite edge computing task unloading and resource allocation method based on deep reinforcement learning

    CN118250750A

  • Low earth orbit satellite network edge computing unloading and resource allocation joint optimization method

    CN118400774A

  • Space-air-ground integrated mobile edge computing unloading and resource allocation optimization method

    CN118524446A

Cited By

  • Computing task unloading optimization method considering multiple performance indexes

    CN119255301A

  • A computational task offloading optimization method considering multiple performance indicators

    CN119255301B