Satellite network resource scheduling optimization method based on heuristic priority sampling DQN

Abstract

The invention discloses a satellite network resource scheduling optimization method based on heuristic priority sampling DQN, which comprises the following steps: initializing DQN network parameters and target network parameters, setting an experience playback buffer area, and defining heuristic priority calculation parameters; at each decision moment, the system selects and executes an optimal action according to a current strategy, observes environment feedback, obtains a next state and an instant reward, and stores experience to a playback buffer area; calculating the priority of each sample in the buffer area, sampling a batch of samples according to the priority for learning, and calculating a target Q value according to a Bellman equation; updating network parameters by using a stochastic gradient descent method; determining how to allocate an upcoming task according to the updated network Q value, and determining whether the task is processed in the current satellite or unloaded to other satellites or ground stations according to the optimized strategy; according to the invention, by integrating heuristic priority sampling and deep reinforcement learning technologies, the resource utilization rate is improved, and the task completion time delay is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a satellite network resource scheduling optimization method based on heuristic priority sampling DQN. Background Art

[0002] Deep Q-Network (DQN) is an algorithm based on deep reinforcement learning. It combines the Q-value function with a deep neural network to achieve action value prediction of high-dimensional input states.

[0003] Traditional DQN is trained by randomly extracting samples from the historical experience pool. Although this method is simple, it still has the following limitations and problems when facing tasks in high-dimensional and complex environments, which restricts its wide application in complex environments:

[0004] 1. Low sample utilization: DQN uses indiscriminate random sampling to train samples in the experience pool. However, random sampling cannot effectively distinguish important samples from ordinary samples, which may lead to inefficient utilization of key experience samples (such as samples with high TD errors); in complex environments, a large number of experience samples are collected and stored, but due to insufficient sample value assessment, the algorithm may fall into an inefficient exploration and learning process in the early stages, resulting in slow convergence.

[0005] 2. Slow convergence: Since the random sampling method of samples fails to effectively focus on samples with higher importance, DQN needs longer time to complete strategy optimization, especially in scenarios with complex tasks and high-dimensional state space. It is difficult for the model to quickly extract useful information from key experiences, thereby delaying strategy updates. In addition, the update lag of the target network may cause training oscillations in the learning process, further affecting model convergence.

[0006] 3. Unstable training: The training process of DQN is easily affected by TD error fluctuations, especially in high-dimensional complex tasks. Unstable error propagation may lead to unstable policy learning. The random sampling mechanism may cause the model to over-train certain samples and ignore other important samples, thereby causing policy deviations and reducing the robustness of the algorithm in different tasks.

[0007] 4. Lack of attention to scarce experience: In many reinforcement learning scenarios, certain states and actions occur less frequently, but they may be important for the task objectives; random sampling makes it difficult to identify and prioritize these scarce experiences, resulting in the model being unable to fully mine key information in the environment.

[0008] These limitations and problems lead to insufficient learning of key information by DQN during training, resulting in low learning efficiency and slow training convergence. In addition, based on this, Schaul et al. proposed a sampling strategy of priority experience replay. Priority Experience Replay (PER) is a reinforcement learning method that can replay important experience samples more frequently by dynamically adjusting sample weights, thereby accelerating the learning process, improving performance, and improving the efficiency of online reinforcement learning. PER universally improves the experience replay mechanism of the DQN algorithm and is suitable for a wide range of tasks. However, for the optimization of specific scenarios, such as satellite network resource scheduling, task scheduling often fails to fully consider the urgency of tasks and the dynamic changes of resources, resulting in inefficient allocation, insufficient efficiency, and excessive response delay.

[0009] Therefore, how to solve the problems of insufficient efficiency and excessive response delay in existing satellite network resource scheduling technology, optimize task scheduling and resource allocation process to improve resource utilization and reduce task completion delay is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0010] In view of this, the present invention provides a satellite network resource scheduling optimization method based on heuristic priority sampling DQN. Through the heuristic priority sampling mechanism, the model convergence speed is accelerated, the strategy adaptability is improved, the average delay is significantly reduced, and the network's response capability to emergencies is enhanced; by introducing a deep Q network based on heuristic priority sampling, the task scheduling and resource allocation process are optimized to improve resource utilization and reduce task completion delay.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] A satellite network resource scheduling optimization method based on heuristic priority sampling DQN includes the following steps:

[0013] S1. Initialize the DQN network parameters and target network parameters, set the experience replay buffer for storing satellite network state transfer data, and define the heuristic priority calculation parameters based on task importance, resource utilization and channel status;

[0014] S2. At each decision moment, the system evaluates all possible resource scheduling schemes based on the current strategy, including the current satellite network status and the delay constraints of the task, and selects the optimal resource allocation or task offloading action; after executing the selected scheduling action, the system observes environmental feedback, including the updated satellite network status and the immediate reward for task completion, and stores the experience in the experience playback buffer;

[0015] S3. Calculate the priority for each scheduling sample stored in the experience replay buffer;

[0016] S4. According to the importance of the task and the scheduling effect, a batch of experiences are sampled from the experience playback buffer according to priority, and the Bellman equation is dynamically updated to calculate the target Q value based on the task delay constraint and resource status;

[0017] S5. Use stochastic gradient descent to optimize network parameters;

[0018] S6. According to the updated network Q value, combined with the satellite resource status and mission delay requirements, the optimized strategy is obtained to determine the upcoming task allocation plan.

[0019] Preferably, in step S2, the current satellite network status includes the computing capacity, channel quality, task queue and current resource occupancy rate of each satellite.

[0020] Preferably, the experience stored in the playback buffer in step S2 includes the real-time status of the satellite network, execution of the selected scheduling action, instant reward for task completion, and updated satellite network status.

[0021] Preferably, in step S3, the priority is calculated as follows:

[0022]

[0023] Among them, ∈ is a small positive number to prevent the denominator from being zero, δ i is the TD error of the current sample;

[0024]

[0025] Among them, r i is the immediate reward, γ is the discount factor, Q(s,α;θ) is the predicted Q value given by the current policy network, Q(s',α';θ - ) is the target network’s optimal Q value estimate for the next state, θ - and θ are the parameters of the target network and the current network respectively.

[0026] Preferably, in step S4, the target Q value is calculated as follows:

[0027]

[0028] The loss function is:

[0029] L(θ)=E[(Q′-Q(s i ,a i ;θ)) 2 ]

[0030] The target Q value directly embeds the task completion rate and delay target into the immediate reward r i In the loss function, the optimization objective is to minimize the mean square error between the predicted Q value Q(s,α;θ) and the target Q value Q′.

[0031] Preferably, the target Q value directly embeds the task completion rate and delay target into the immediate reward r i That is, to meet the satellite resource scheduling optimization goal, specifically to maximize the mission completion rate while meeting the mission delay requirements;

[0032]

[0033] Among them, D i is the task completion delay, C i is the completion rate of the task, α and β are weight factors that weigh the delay and completion rate.

[0034] Preferably, the content of updating the network parameters is: using a neural network composed of a convolutional layer and a fully connected layer to approximate the Q value function, and copying the value of the current network parameter to the target network parameter every fixed number of steps.

[0035] Preferably, in step S6, the optimized strategy gives priority to meeting the task completion rate and delay requirements, maximizes satellite network resource utilization and reduces task processing delays; the task allocation plan of step S6 is specifically: whether the task is completed on the current satellite, offloaded to other satellites for processing, or transmitted to a ground station.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a satellite network resource scheduling optimization method based on heuristic priority sampling DQN.

[0037] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the satellite network resource scheduling optimization method based on heuristic priority sampling DQN is implemented.

[0038] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a satellite network resource scheduling optimization method based on heuristic priority sampling DQN, which has the following beneficial effects:

[0039] (1) Improved resource utilization efficiency: The present invention can more effectively utilize limited satellite network resources by dynamically adjusting the priority of experience playback. The heuristic priority sampling mechanism ensures that more important training samples are frequently learned, which accelerates the learning process and improves the quality of resource scheduling decisions.

[0040] (2) Reduced task completion delay: The present invention optimizes the processing of delay-sensitive tasks during the training process, thereby ensuring that the performance requirements of delay-sensitive applications are met, realizing resource allocation and computational offloading strategies for precise control tasks, and significantly reducing communication and processing delays in satellite networks;

[0041] (3) Enhanced network adaptability and robustness: The deep learning model used in the present invention can automatically learn and adapt to changes in the satellite network environment, such as channel status and network load fluctuations. This adaptability enables the network to exhibit higher robustness when facing different operating conditions and requirements;

[0042] (4) Reduced energy consumption: In satellite networks, the optimization of resource allocation and computing tasks not only improves efficiency but also reduces the energy consumption of the system. The present invention helps to improve energy efficiency by reducing unnecessary data transmission and computing processes.

[0043] In general, the present invention proposes an effective resource scheduling and computational offloading strategy, which significantly improves the performance of satellite networks by integrating heuristic priority sampling and deep reinforcement learning technology, and solves the problems of low resource utilization and high response delay in the prior art; in addition, the present invention also enhances the adaptability and robustness of the network in a dynamic environment, bringing innovative solutions to the field of satellite communications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A schematic diagram of a satellite network resource scheduling optimization method based on heuristic priority sampling DQN provided by the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The embodiment of the present invention discloses a satellite network resource scheduling optimization method based on heuristic priority sampling DQN, comprising the following steps:

[0048] S1. Initialize DQN network parameters θ and target network parameters θ - , these parameters determine how the network predicts the Q value of each state-action pair; set the experience replay buffer used to store the satellite network state transition data, including the state s t , the action taken is a t , the reward r t , and the next state s t+1 ; and define heuristic priority calculation parameters based on task importance, resource utilization and channel status, which will dynamically adjust the storage priority of experience based on TD error;

[0049] S2. At each decision time t, the system makes decisions based on the current strategy π(a|s t ; θ), including the current satellite network status and the delay constraint of the task, evaluates all possible resource scheduling schemes and selects the optimal resource allocation or task offloading action a t ; Execute the selected scheduling action a t , observe environmental feedback, including updated satellite network status s t+1 and instant rewards for completing tasks t , and store the experience (s t ,a t ,r t ,s t+1 ) to the playback buffer;

[0050] S3. Calculate the priority P(i) for each scheduling sample i stored in the experience replay buffer;

[0051] S4. According to the calculated sample priority, that is, according to the importance of the task and the scheduling effect, a batch of experience {(s i ,a i ,r i ,s i ')}, used to train the network, dynamically update the Bellman equation to calculate the target Q value based on the task delay constraints and resource status;

[0052] S5. Update network parameters using stochastic gradient descent method;

[0053] S6. According to the updated network Q value Q(s,α;θ), combined with the satellite resource status and mission delay requirements, the optimized strategy π(a|s t ;θ), determine the upcoming task allocation plan.

[0054] In order to further implement the above technical solution, in step S2, the current satellite network status includes the computing power, channel quality, task queue and current resource occupancy rate of each satellite.

[0055] To further implement the above technical solution, the experience stored in the playback buffer in step S2 includes the real-time status of the satellite network, the execution of the selected scheduling action, the immediate reward for task completion, and the updated satellite network status.

[0056] In this embodiment, heuristic priority sampling is an improved experience replay mechanism for dynamically adjusting the sampling priority of samples stored in the experience replay buffer. It mainly evaluates the importance of each sample based on its time difference error, which is the difference between the predicted Q value and the target Q value. The larger the value, the greater the amount of information that can be brought by learning the sample, so a higher sampling priority should be given.

[0057] In order to further implement the above technical solution, in step S3, the priority is calculated as follows:

[0058]

[0059] Among them, v is a small positive number to prevent the denominator from being zero, δ i is the TD error of the current sample;

[0060]

[0061] Among them, r i is the immediate reward, γ is the discount factor, Q(s,α;θ) is the predicted Q value given by the current policy network, Q(s',α';θ - ) is the target network’s optimal Q value estimate for the next state, θ - and θ are the parameters of the target network and the current network respectively.

[0062] In order to further implement the above technical solution, in step S4, the target Q value is calculated as follows:

[0063]

[0064] The loss function is:

[0065] L(θ)=E[(Q′-Q(s i ,a i ;θ)) 2 ]

[0066] The target Q value directly embeds the task completion rate and delay target into the immediate reward r i In the loss function, the optimization objective is to minimize the mean square error between the predicted Q value Q(s,α;θ) and the target Q value Q′.

[0067] In order to further implement the above technical solution, the target Q value directly embeds the task completion rate and delay target into the immediate reward r iThat is, to meet the satellite resource scheduling optimization goal, specifically to maximize the mission completion rate while meeting the mission delay requirements;

[0068]

[0069] Among them, D i is the task completion delay, C i is the completion rate of the task, α and β are weight factors that weigh the delay and completion rate.

[0070] In this embodiment, tasks are obtained by the task arrival model, which is the basis for resource scheduling and computation offloading decisions. Assuming that tasks arrive randomly according to the Poisson process, this model can effectively simulate the randomness of task arrival in the real world. Each task T i Define a tuple containing computing requirements and latency constraints, denoted as T i =(C i ,D i ), where C i is the computational requirements of the task, including the number of CPU cycles and memory size required, D i is the delay constraint, which is the latest time for task completion.

[0071] In order to further implement the above technical solution, the content of updating the network parameters is: use a neural network composed of convolutional layers and fully connected layers to approximate the Q-value function, and copy the values ​​of the current network parameters to the target network parameters every fixed number of steps. This can fix the target value and avoid target deviation during training, so as to improve training stability.

[0072] In order to further implement the above technical solution, in step S6, the optimized strategy gives priority to meeting the task completion rate and delay requirements, maximizes the satellite network resource utilization and reduces the task processing delay; the task allocation plan of step S6 is specifically: whether the task is completed on the current satellite, offloaded to other satellites for processing, or transmitted to the ground station.

[0073] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a satellite network resource scheduling optimization method based on heuristic priority sampling DQN.

[0074] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the satellite network resource scheduling optimization method based on heuristic priority sampling DQN is implemented.

[0075] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0076] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A satellite network resource scheduling optimization method based on heuristic priority sampling DQN, characterized in that: The following steps are involved: S1. Initialize the DQN network parameters and target network parameters, set the experience replay buffer for storing satellite network state transfer data, and define the heuristic priority calculation parameters based on task importance, resource utilization and channel status; S2. At each decision moment, the system evaluates all possible resource scheduling schemes based on the current strategy, including the current satellite network status and the delay constraints of the task, and selects the optimal resource allocation or task offloading action; after executing the selected scheduling action, the system observes environmental feedback, including the updated satellite network status and the immediate reward for task completion, and stores the experience in the experience playback buffer; S3. Calculate the priority for each scheduling sample stored in the experience replay buffer; S4. According to the importance of the task and the scheduling effect, a batch of experiences are sampled from the experience playback buffer according to priority, and the Bellman equation is dynamically updated to calculate the target Q value based on the task delay constraint and resource status; S5. Use stochastic gradient descent to optimize network parameters; S6. According to the updated network Q value, combined with the satellite resource status and mission delay requirements, the optimized strategy is obtained to determine the upcoming task allocation plan.

2. According to claim 1, a satellite network resource scheduling optimization method based on heuristic priority sampling DQN is characterized in that: In step S2, the current satellite network status includes the computing capacity, channel quality, task queue and current resource occupancy rate of each satellite.

3. According to claim 1, a satellite network resource scheduling optimization method based on heuristic priority sampling DQN is characterized in that: The experience stored in the playback buffer in step S2 includes the real-time status of the satellite network, the execution of the selected scheduling action, the immediate reward for task completion, and the updated satellite network status.

4. According to claim 1, a satellite network resource scheduling optimization method based on heuristic priority sampling DQN is characterized in that: In step S3, the priority is calculated as follows: Among them, ∈ is a small positive number to prevent the denominator from being zero, δ i is the TD error of the current sample; Among them, r i is the immediate reward, γ is the discount factor, Q(s,α;θ) is the predicted Q value given by the current policy network, Q(s',α';θ - ) is the target network’s optimal Q value estimate for the next state, θ - and θ are the parameters of the target network and the current network respectively.

5. According to claim 1, a satellite network resource scheduling optimization method based on heuristic priority sampling DQN is characterized in that: In step S4, the target Q value is calculated as follows: The loss function is: L(θ)=E[(Q′-Q(s i ,a i ;i)) 2 ] The target Q value directly embeds the task completion rate and delay target into the immediate reward r i In the loss function, the optimization objective is to minimize the mean square error between the predicted Q value Q(s,α;θ) and the target Q value Q′.

6. According to claim 5, a satellite network resource scheduling optimization method based on heuristic priority sampling DQN is characterized in that: The target Q value directly embeds the task completion rate and delay target into the immediate reward r i That is, to meet the satellite resource scheduling optimization goal, specifically to maximize the mission completion rate while meeting the mission delay requirements; Among them, D i is the task completion delay, C i is the completion rate of the task, α and β are weight factors that weigh the delay and completion rate.

7. The satellite network resource scheduling optimization method based on heuristic priority sampling DQN according to claim 1 is characterized in that: The content of updating the network parameters is: use a neural network composed of convolutional layers and fully connected layers to approximate the Q-value function, and copy the values ​​of the current network parameters to the target network parameters every fixed number of steps.

8. The satellite network resource scheduling optimization method based on heuristic priority sampling DQN according to claim 1 is characterized in that: In step S6, the optimized strategy gives priority to meeting the task completion rate and latency requirements, maximizes satellite network resource utilization and reduces task processing delays; the task allocation plan of step S6 is specifically: whether the task is completed on the current satellite, offloaded to other satellites for processing, or transmitted to the ground station.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the satellite network resource scheduling optimization method based on heuristic priority sampling DQN as described in any one of claims 1 to 8 is implemented.

10. A processing terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the satellite network resource scheduling optimization method based on heuristic priority sampling DQN as described in any one of claims 1 to 8 is implemented.

Images