A high-low orbit satellite dynamic shunting method and system based on deep reinforcement learning

Abstract

The present application relates to the technical field of satellite offloading, in particular to a deep reinforcement learning-based high-low orbit satellite dynamic offloading method and system, which specifically comprises state sensing and demand acquisition, link state time sequence prediction, terminal local reinforcement learning decision-making, differentiated service offloading execution, strategy smoothing verification and feedback optimization. Through the combination of terminal-side edge intelligent decision-making architecture and deep reinforcement learning-based basic decision-making, the offloading decision can be sunk to the terminal local, reducing the dependence on the ground control center and the on-board processing node, greatly reducing the end-to-end decision-making delay and effectively avoiding the risk of strategy paralysis caused by the interruption of the control link.

Description

Technical Field

[0001] This invention relates to the field of satellite diversion technology, and more specifically, to a method and system for dynamic diversion of high and low orbit satellites using deep reinforcement learning. Background Technology

[0002] With the rapid development of satellite communication networks, high- and low-orbit satellite collaborative networking has become an important technical path for building a globally seamless, integrated air-space-ground information network.

[0003] In the prior art, document CN121887263A discloses "A Task Scheduling Method for Low-Earth Orbit Satellite Networks Based on Hybrid Reinforcement Learning." This method prioritizes tasks using a multi-factor dynamic weighted ranking strategy, constructs a multi-dimensional feature vector including node resource status, task characteristics, and historical performance, and employs an A2C reinforcement learning architecture to optimize scheduling decisions. This invention enables the system to intelligently adjust its scheduling focus based on real-time load in a dynamically changing satellite network environment, effectively improving task success rate. Particularly under high load, it prioritizes high-priority tasks, reduces average task latency, and optimizes satellite energy consumption.

[0004] The document CN121308819A discloses "A Segmented Satellite Routing Method and System Based on Multi-Agent Reinforcement Learning". The steps are as follows: Step 1, acquire satellite network structure data and construct a static topology model; Step 2, cluster the satellite network based on the static topology model; Step 3, based on the clustering results, obtain the optimal edge node and formulate inter-cluster routing strategies; Step 4, construct a satellite intelligent network for each satellite based on the static topology model and obtain the corresponding satellite state information; Step 5, formulate intra-cluster routing strategies based on the optimal edge node and satellite state information.

[0005] In summary, while existing technologies reduce on-board computational complexity through clustering and segmented routing, and improve task scheduling efficiency by combining hybrid reinforcement learning with heuristic strategies, enabling intelligent adjustment of scheduling focus based on real-time load to effectively improve task success rate, reduce average latency, and optimize satellite energy consumption, and enhance routing stability through latitude-based dynamic clustering and multi-path rerouting, they suffer from several drawbacks. These include a centralized decision-making architecture leading to a lack of terminal autonomy, high end-to-end decision-making latency, and weak anti-interference capabilities. Furthermore, resource allocation is coarse-grained, relying solely on bandwidth or latency as the primary metric for traffic allocation, lacking joint optimization based on service priority and multi-dimensional link states, which can easily lead to resource mismatch. Decisions are also based solely on instantaneous states, lacking prediction of time-varying trends in low-Earth orbit links, resulting in frequent switching during overhead passages causing service jitter. Finally, fixed optimization targets prevent dynamic adjustment of protection priorities based on electromagnetic interference, leading to insufficient reliability of core services in extreme environments. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for dynamic diversion of high and low orbit satellites using deep reinforcement learning, in order to solve the problems mentioned in the background art.

[0007] This invention is achieved through the following technical solution: Firstly, a deep reinforcement learning-based method for dynamic satellite traffic allocation in high and low orbits includes: Real-time acquisition of instantaneous basic parameters of high-orbit and low-orbit satellite links, and acquisition of priority labels and real-time traffic requirements of each service data stream currently to be scheduled; Retrieve the historical link state sequence within a preset period, generate high- and low-orbit link state prediction sequences for multiple future decision periods through a lightweight time series prediction model, and concatenate the real-time state parameters with the state prediction sequences to form the input state set of the deep reinforcement learning model. The deep reinforcement learning model dynamically adjusts the priority weights of each service flow based on the electromagnetic interference level of the current link, outputs the traffic allocation ratio of each service flow on the high-speed and low-speed rail links, forms a routing policy matrix, and performs service diversion scheduling based on the routing policy matrix. The routing strategy matrix of the current cycle is compared with that of the previous cycle. The traffic splitting strategy is verified and implemented after combining the preset smoothing constraint rules. The link status and service transmission results of this round of transmission are stored in the historical dataset.

[0008] Preferably, the dynamic adjustment of the priority weights of each service flow includes:

[0009] In the formula, Let i be the dynamic priority weight of the i-th service flow under the current electromagnetic interference level. The normalized electromagnetic interference level for the current decision-making cycle. Let the base priority weight be the i-th business flow. is the weight adjustment coefficient for the i-th business flow.

[0010] Preferably, it also includes calculating the multi-dimensional QoS utility value of a single service flow as the basic calculation unit for overall utility evaluation:

[0011] In the formula, Let i be the multidimensional QoS utility value of the i-th service flow. The actual transmission bandwidth obtained by the i-th service flow. The actual round-trip delay obtained by the i-th service flow is Let be the actual bit error rate obtained by the i-th service flow. This represents the bandwidth compliance threshold for the i-th type of service. Let be the latency tolerance threshold for the i-th type of service. The utility coefficient is the one representing bandwidth revenue. The utility coefficient is the dimension of delay loss. This is the utility coefficient for the bit error loss dimension.

[0012] Preferably, the deep reinforcement learning model is a deep Q-network model, and value iteration and optimal policy reasoning are performed based on the deep Q-network model:

[0013] In the formula, Let be the optimal action value for performing action a in state s. This is the current state. For the current action, For the state at the next moment, This is an optional action for the next moment. The immediate reward obtained by performing action a in state s. As a discount factor, For the set of all selectable actions, The Bellman equation is a Q-function.

[0014] Preferably, the priority labels of each service data stream are divided into three levels, including the highest priority corresponding to emergency control instructions, the second highest priority corresponding to video transmission, and the ordinary priority corresponding to sensor telemetry data; the link status parameters are sampled and obtained by the channel sensing module on the terminal side at a fixed decision period, and the sampling period is consistent with the decision period.

[0015] Preferably, the comparison of the routing strategy matrix differences between the current period and the previous period includes: Retrieve the routing strategy matrix that was finally implemented in the previous decision cycle, and perform element-wise difference calculation with the initial routing strategy matrix output by the current deep reinforcement learning model. Quantize the overall difference between the two matrices using the norm, and compare the calculated difference value with the preset smoothing threshold. If the difference value does not exceed the threshold, the current strategy is determined to meet the smoothing constraint requirements and is executed; if the difference value exceeds the threshold, the allocation ratio of ordinary priority business and second-highest priority business is compressed successively according to the business priority from low to high.

[0016] Preferably, it also includes the quantitative calculation of instant rewards, including:

[0017] In the formula, For the current decision-making cycle in state Next action Instant rewards received This represents the current state of the decision-making cycle. This is the routing strategy matrix for the current decision-making cycle. This is the routing strategy matrix from the previous decision cycle. The penalty coefficient for strategy switching. This represents the total QoS utility value of the system under the current state when implementing the current policy.

[0018] Preferably, the complete set of input states corresponding to this round of decision-making, the verified final routing strategy matrix, the calculated instant reward value, and the actual transmission results such as the actual transmission bandwidth, round-trip delay, bit error rate, and actual delivery rate of various services of the high-speed and low-speed links obtained after the end of this decision-making cycle are associated and encapsulated according to a unified timestamp and written into the local historical dataset.

[0019] Preferably, the method also includes offline iterative training of the deep Q-network model based on local historical datasets, extracting historical decision samples in chronological order and substituting them into the value iteration process to update the network weights, gradually optimizing the decision accuracy and stability of the model under different electromagnetic interference levels and link time-varying scenarios, keeping the overall decision framework of the model unchanged during the training process, only updating the internal parameters of the network, and replacing the original model with the model after training convergence.

[0020] Secondly, the present invention also provides a deep reinforcement learning-based dynamic satellite traffic allocation system for high and low orbit orbits, used to execute the aforementioned deep reinforcement learning-based dynamic satellite traffic allocation method, comprising: The data processing module is configured to collect instantaneous basic parameters of high-orbit and low-orbit satellite links in real time, and simultaneously obtain the priority labels and real-time traffic requirements of each service data stream to be scheduled. It retrieves historical link state sequences within a preset period, generates high-orbit and low-orbit link state prediction sequences for multiple future decision periods through a lightweight time series prediction model, and concatenates the real-time state parameters with the state prediction sequences to form the input state set of the deep reinforcement learning model. The deep reinforcement learning model dynamically adjusts the priority weights of each service stream according to the electromagnetic interference level of the current link, outputs the traffic allocation ratio of each service stream on the high-orbit and low-orbit links, forms a routing strategy matrix, and performs service diversion scheduling according to the routing strategy matrix. The execution judgment module is configured to compare the difference between the routing strategy matrix of the current period and the previous period, verify the traffic splitting strategy in combination with the preset smoothing constraint rules, and then execute it. The link status and service transmission results during this round of transmission are stored in the historical dataset.

[0021] The technical solution of the present invention has at least the following advantages and beneficial effects: This invention combines edge-based intelligent decision-making architecture on the terminal side with deep reinforcement learning-based decision-making, enabling traffic diversion decisions to be made locally on the terminal. This reduces reliance on ground control centers and on-board processing nodes, significantly lowers end-to-end decision latency, and effectively avoids the risk of policy paralysis caused by control link interruptions. Furthermore, by using link time-varying prediction for advance guidance, the model possesses forward planning capabilities, enabling it to adjust traffic diversion strategies in advance based on link change trends. This significantly reduces frequent switching during low-Earth orbit overpasses, ensuring the continuity of service transmission.

[0022] Meanwhile, by adaptively adjusting priority weights, the optimization target can be dynamically changed with the electromagnetic environment. In strong interference scenarios, core businesses are prioritized, and under normal circumstances, the overall resource utilization rate is improved, thus achieving precise assurance of differentiated service quality. Furthermore, by using policy switching smoothing constraints, policy oscillations are suppressed from the root cause of optimization, further reducing service latency jitter. This enables refined, adaptive, and highly stable dynamic traffic splitting in complex electromagnetic environments, improving the overall efficiency of terminal communication and the reliability of core services. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the process flow for a deep reinforcement learning-based dynamic satellite diversion method for high and low orbit satellites according to the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0026] The module division in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed.

[0027] The independently described modules or sub-modules may or may not be physically separated; they may be implemented in software or hardware, and some modules or sub-modules may be implemented in software, with the processor calling the software to implement the function of these modules or sub-modules, while other modules or sub-modules may be implemented in hardware, such as through hardware circuits. Furthermore, some or all of the modules can be selected to achieve the purpose of this application's solution according to actual needs.

[0028] Please refer to Figure 1 This invention provides a deep reinforcement learning-based method for dynamic traffic allocation between high and low Earth orbit satellites, comprising: S1: Real-time acquisition of instantaneous basic parameters of high-orbit and low-orbit satellite links, and acquisition of priority labels and real-time traffic requirements of each service data stream currently to be scheduled; Real-time acquisition of instantaneous available bandwidth, round-trip delay, and bit error rate status parameters of high-orbit and low-orbit satellite links; at the same time, acquisition of priority tags and real-time traffic requirements of each service data stream currently to be scheduled. Specifically, the priority labels of the business data streams are divided into three levels according to the business attributes: the highest priority corresponding to emergency control instructions, the second highest priority corresponding to high-definition video transmission, and the ordinary priority corresponding to sensor telemetry data; the link status parameters are sampled and obtained by the channel sensing module on the terminal side at a fixed decision period, and the sampling period is consistent with the decision period.

[0029] At the beginning of each decision cycle, the terminal sends probe data frames of fixed length to both high-orbit and low-orbit satellites. It records the sending timestamp of each frame and the receiving timestamp of the corresponding acknowledgment frame. The difference between the two is the round-trip delay of the corresponding link. The ratio of the total amount of data successfully transmitted in a single decision cycle to the cycle duration is calculated to obtain the instantaneous available bandwidth. At the same time, by comparing the cyclic redundancy check bits of the received data frames with the preset check value, the ratio of the number of error bits to the total number of transmitted bits is calculated to obtain the real-time bit error rate. After the three types of parameters are calculated, they are temporarily stored in the local cache aligned with the same timestamp. The priority matching and traffic statistics of the scheduled services are achieved by the terminal reading the header type identifier of the data frames to be sent in the uplink data buffer and matching them with the corresponding priority level. Among them, the data frames carrying the emergency control identifier correspond to the highest priority, the data frames carrying the high-definition video identifier correspond to the second highest priority, and the data frames carrying the sensor telemetry identifier correspond to the normal priority. At the same time, the total amount of data to be transmitted in each decision cycle is accumulated according to the priority category to obtain the real-time traffic demand of various services. No manual pre-configuration of service levels is required throughout the process. In addition, the terminal uses a local unified clock to synchronously trigger link status sampling and service traffic statistics. The sampling trigger time is fully aligned with the start time of the decision cycle to avoid the mismatch between status parameters and service requirements due to time deviation. The sampling frequency corresponds one-to-one with the decision cycle, and each decision cycle only performs a complete status collection and requirement statistics once. After the collection and statistics are completed, the terminal performs anomaly verification on the acquired link status parameters and business traffic data. Parameter values ​​that exceed the physical reasonable range are replaced with valid values ​​from the previous period to avoid abnormal data from interfering with subsequent decision output. The data after verification is used as the basic input data for the current period and awaits retrieval.

[0030] S2: Retrieve the historical link state sequence within the preset period, generate the high and low track link state prediction sequence for multiple future decision periods through a lightweight time series prediction model, and concatenate the real-time state parameters with the state prediction sequence to form the input state set of the deep reinforcement learning model. Retrieve the historical link state sequence within a preset period, generate high- and low-orbit link state prediction sequences for multiple future decision periods through a lightweight time series prediction model, and concatenate the real-time state parameters with the state prediction sequences to form the input state set of the deep reinforcement learning model. Specifically, a lightweight time-series prediction model is deployed locally on the terminal, outputting multi-step link state change trends based on historical link state sequences. The generated prediction sequence is then concatenated with real-time state parameters to form the model input state set.

[0031] First, the terminal retrieves the high- and low-rail link status parameters corresponding to a preset number of consecutive decision cycles from the historical dataset stored locally. These parameters are arranged into a time series of equal length in chronological order. The different types of status parameters within the series are then normalized and mapped to eliminate dimensional differences before being used as the input basis for prediction calculation. Then, the completed historical time series is input into the lightweight time series prediction model. Based on the fluctuation pattern and periodic change characteristics of the link status in the historical period, the model successively predicts the trend of link status changes in each subsequent decision cycle. The entire prediction process is completed by calling the local computing resources of the terminal and does not require data interaction with the ground control center or on-board nodes. The model then outputs predicted values ​​of available bandwidth, round-trip delay, and bit error rate for high-speed and low-speed rail links for multiple decision cycles in the future. These values ​​are arranged sequentially according to time steps to form a complete state prediction sequence. After the output is completed, the starting value of the prediction sequence is checked against the current real-time state parameters to correct abnormal prediction points with obvious jumps and ensure the smoothness of trend transition. Finally, the link state parameters collected in real time during the current period are used as the current time node of the time series and concatenated with the generated multi-step prediction sequence along the time dimension to form a complete set of input states that simultaneously contains the instantaneous real state and future trend prediction. This set is then directly passed to the input layer of the deep reinforcement learning decision model for subsequent inference calculation.

[0032] S3: The deep reinforcement learning model dynamically adjusts the priority weights of each service flow based on the electromagnetic interference level of the current link, outputs the traffic allocation ratio of each service flow on the high-speed and low-speed rail links, forms a routing policy matrix, and performs service diversion scheduling based on the routing policy matrix. The input state set is input into a deep reinforcement learning decision model deployed locally on the terminal. The model dynamically adjusts the priority weight of each service flow according to the electromagnetic interference level of the current link, and outputs the traffic allocation ratio of each service flow on the high-speed and low-speed links through inference, forming a routing strategy matrix. Specifically, the deep reinforcement learning decision model adopts a deep Q-network architecture and is deployed on the edge computing unit local to the satellite terminal. The inference process does not rely on the ground control center or on-board processing nodes. In the routing policy matrix output by the model, each element corresponds to the traffic allocation ratio of a single service flow on a single link, with a value range of 0 to 1. The sum of the allocation ratios of the same service flow on the high-Earth orbit and low-Earth orbit links is 1.

[0033] The model dynamically updates the priority weights of each service flow based on the electromagnetic interference level generated in real time by the terminal. The weight update adopts an exponential adjustment form, as shown in the following expression:

[0034] In the formula, Let i be the dynamic priority weight of the i-th service flow under the current electromagnetic interference level. The normalized electromagnetic interference level for the current decision-making cycle. Let the base priority weight be the i-th business flow. is the weight adjustment coefficient for the i-th business flow.

[0035] The terminal first normalizes and maps the real-time collected link signal-to-interference-plus-noise ratio and bit error rate fluctuation amplitude to obtain the current electromagnetic interference level value in the range of 0 to 1. Then, it combines the pre-configured baseline priority weights and corresponding weight adjustment coefficients of various services to calculate the dynamic priority weights adapted to the current electromagnetic environment. The higher the electromagnetic interference intensity, the more significant the increase in the weight of high-priority emergency services, while the weight of ordinary services is appropriately reduced in sync. The adjusted dynamic weights directly participate in the subsequent utility calculation process. The multi-dimensional QoS utility quantification of a single service flow is the basic calculation unit for evaluating the overall system utility. It uses a combined utility approach to quantify bandwidth gains, latency losses, and bit error losses, as shown below:

[0036] In the formula, Let i be the multidimensional QoS utility value of the i-th service flow. The actual transmission bandwidth obtained by the i-th service flow. The actual round-trip delay obtained by the i-th service flow is Let be the actual bit error rate obtained by the i-th service flow. This represents the bandwidth compliance threshold for the i-th type of service. Let be the latency tolerance threshold for the i-th type of service. The utility coefficient is the one representing bandwidth revenue. The utility coefficient is the dimension of delay loss. This is the utility coefficient for the bit error loss dimension.

[0037] The terminal matches the corresponding utility coefficient and threshold parameters according to the priority label of the service flow, substitutes the actual transmission bandwidth, round-trip delay and bit error rate allocated to the service flow under the current diversion strategy into the formula to calculate, positively accumulates the utility gain brought by the bandwidth resource supply, and negatively deducts the utility loss caused by excessive delay and increased bit error rate, and finally obtains the independent utility value of a single service flow. Furthermore, the total QoS utility of the system is obtained by integrating the utility values ​​of all service flows through a weighted summation method. The expression for the total system utility function is:

[0038] In the formula, This represents the total QoS utility value of the system. Given the total number of service flows to be scheduled, the terminal multiplies the dynamic priority weight corresponding to each service flow with the calculated single service utility value, and then sums the weighted utility results of all service flows to be scheduled to obtain the total system utility value corresponding to the current candidate diversion strategy. This value is directly used as the core component of the reinforcement learning immediate reward and is used to quantitatively evaluate the overall merits of the current diversion strategy. Finally, the deep Q-network model follows the Bellman optimality equation to complete value iteration and optimal policy reasoning, expressed as:

[0039] In the formula, Let be the optimal action value for performing action a in state s. This is the current state. For the current action, For the state at the next moment, This is an optional action for the next moment. The immediate reward obtained by performing action a in state s. As a discount factor, For the set of all selectable actions, The Bellman equation is a Q-function.

[0040] The model takes the set of input states, which includes real-time link status and future prediction sequences, as the current state, and all feasible routing policy matrices as the set of optional actions. Iteratively calculates the optimal action value for each state-action pair by combining the instant reward value and discount factor, and finally selects the routing policy matrix that maximizes the total utility of the system as the output result. The entire inference calculation is completed in the local edge computing unit of the terminal, without relying on the ground control center or on-board processing nodes.

[0041] Based on the routing strategy matrix, service diversion and scheduling are performed, scheduling high-priority control command services to high-orbit satellite links for transmission, and scheduling high-bandwidth video and batch telemetry services to low-orbit satellite links for transmission; Specifically, during the service diversion and scheduling process, high-orbit satellite links prioritize the transmission of high-priority services, while low-orbit satellite links, based on their own channel quality status, prioritize the transmission of high-volume services.

[0042] First, the terminal parses the routing policy matrix output by the deep reinforcement learning model, reads the traffic allocation ratio of each service flow on the high and low track links in descending order of service priority, and matches it with the three types of service data queues in the uplink data buffer. It generates a corresponding link transmission byte quota for each queue. The highest priority emergency instruction queue locks the transmission quota of the high track link first. After the quota is locked, other low priority services are not allowed to occupy the link resources. This ensures the transmission priority of high priority services from the resource allocation level. Then, the terminal performs frame-by-frame scheduling and transmission according to the link quota of each queue. For high-priority emergency control services, the entire service is scheduled to the high-orbit link for transmission according to the strategy. The reliability of command transmission is ensured by relying on the wide coverage and relatively stable link characteristics of high-orbit satellites. For high-definition video and sensor telemetry services with large bandwidth, when the channel quality of the low-orbit link meets the preset transmission conditions, the service is scheduled to the low-orbit link for transmission according to the allocation ratio. When the allocation ratio of a single service stream is between 0 and 1, it is split into two links for parallel transmission according to the ratio. The large bandwidth characteristics of the low-orbit link are fully utilized to improve the overall transmission efficiency and achieve the coordinated transmission effect of high-orbit coverage and low-orbit capacity supplementation. Finally, during the transmission process of a single decision cycle, the terminal performs real-time statistics on the actual carrying traffic and service transmission status of the two links. When the actual available bandwidth of a certain link fluctuates slightly in a short period of time, it only fine-tunes the actual transmission rate of each service flow within the same priority service, without adjusting resource allocation across priorities, to ensure that the allocation ratio of high-priority services does not shift. At the same time, it synchronously records the actual transmission traffic of each service flow and the actual carrying data of the link in this cycle to the local cache, providing real operational data support for the status collection and model optimization of subsequent decision cycles.

[0043] S4: Compare the differences between the routing strategy matrix of the current period and the previous period, verify the traffic splitting strategy in combination with the preset smoothing constraint rules, and then implement it. Store the link status and service transmission results in the current transmission process into the historical dataset.

[0044] By comparing the differences between the routing strategy matrices of the current cycle and the previous cycle, and combining the preset smoothing constraint rules to verify the traffic splitting strategy, the strategy is implemented and executed. The link status and service transmission results during this round of transmission are stored in the historical dataset for continuous optimization of the deep reinforcement learning model. Specifically, the smoothing constraint rule corrects traffic splitting strategies that exceed a set range of difference by comparing the degree of difference between two adjacent periodic routing strategy matrices, thus constraining the magnitude of changes in traffic splitting strategies.

[0045] The terminal first retrieves the routing policy matrix that was finally executed in the previous decision cycle from its local cache, and performs element-by-element difference calculation with the initial routing policy matrix output by the current deep reinforcement learning model. The Frobenius norm is used to quantify the overall difference between the two matrices, and the calculated difference value is compared with a preset smoothing threshold. If the difference value does not exceed the threshold, the current policy is determined to meet the smoothing constraint requirements and can be directly executed. If the difference value exceeds the threshold, the allocation ratio of ordinary priority services and the second highest priority services is compressed in order of service priority from low to high, and the link allocation ratio of high priority emergency services is locked first. After the correction is completed, the difference is re-verified until the overall matrix difference meets the preset threshold requirements. This constrains the drastic fluctuations of the traffic diversion policy from the execution and landing level, and avoids frequent policy oscillations that cause service transmission jitter. The quantitative calculation of immediate rewards is the core evaluation criterion supporting the continuous iterative optimization of reinforcement learning models. The complete immediate reward adopts a calculation form that includes a policy switching penalty term, specifically:

[0046] In the formula, For the current decision-making cycle in state Next action Instant rewards received This represents the current state of the decision-making cycle. This is the routing strategy matrix for the current decision-making cycle. This is the routing strategy matrix from the previous decision cycle. The penalty coefficient for strategy switching. This represents the total QoS utility value of the system under the current state when implementing the current policy.

[0047] The terminal takes the current period's link status and service demand set as the status input, and the final routing policy matrix after verification and correction as the execution action. First, it substitutes the total QoS utility value corresponding to the current policy into the system's total utility function to calculate the total QoS utility value corresponding to the current policy as the main body of positive reward. Then, it combines the Frobenius norm difference of the routing policy matrix of adjacent periods and the preset penalty coefficient to calculate the policy switching penalty value. After deducting the penalty value from the positive utility value, the final real-time reward value is obtained. This value can simultaneously reflect the comprehensive service quality and policy stability of the current traffic diversion policy. Next, the terminal associates and encapsulates the complete set of input states corresponding to this round of decision-making, the verified final routing strategy matrix, the calculated instant reward value, and the actual transmission results such as the actual transmission bandwidth, round-trip delay, bit error rate, and actual delivery rate of various services of the high-speed and low-speed links obtained after the end of this decision-making cycle, and writes them into the local historical dataset after being associated with a unified timestamp. During the storage process, duplicate timestamps and abnormal values ​​that exceed the physical reasonable range are automatically filtered to ensure the temporal continuity and sample validity of historical data, and to provide standardized sample data support for subsequent model iteration training. Finally, during idle periods of service transmission, the terminal performs offline iterative training on the deep Q-network model based on local historical datasets. Historical decision samples are extracted in chronological order and substituted into the value iteration process to update network weights. The decision accuracy and stability of the model are gradually optimized under different electromagnetic interference levels and link time-varying scenarios. The overall decision framework of the model remains unchanged during the training process, and only the internal parameters of the network are updated. The model after training convergence automatically replaces the original model and is used for traffic inference in subsequent decision cycles, so that the model can continuously adapt to the changing spatial communication environment and differentiated service transmission requirements.

[0048] In one embodiment, the priority labels of each service data stream are divided into three levels, including the highest priority corresponding to emergency control instructions, the second highest priority corresponding to video transmission, and the ordinary priority corresponding to sensor telemetry data; the link status parameters are sampled and obtained by the channel sensing module on the terminal side at a fixed decision period, and the sampling period is consistent with the decision period.

[0049] Secondly, the present invention also provides a deep reinforcement learning-based dynamic satellite traffic allocation system for high and low orbit orbits, used to execute the aforementioned deep reinforcement learning-based dynamic satellite traffic allocation method, comprising: The data processing module is configured to collect instantaneous basic parameters of high-orbit and low-orbit satellite links in real time, and simultaneously obtain the priority labels and real-time traffic requirements of each service data stream to be scheduled. It retrieves historical link state sequences within a preset period, generates high-orbit and low-orbit link state prediction sequences for multiple future decision periods through a lightweight time series prediction model, and concatenates the real-time state parameters with the state prediction sequences to form the input state set of the deep reinforcement learning model. The deep reinforcement learning model dynamically adjusts the priority weights of each service stream according to the electromagnetic interference level of the current link, outputs the traffic allocation ratio of each service stream on the high-orbit and low-orbit links, forms a routing strategy matrix, and performs service diversion scheduling according to the routing strategy matrix. The execution judgment module is configured to compare the difference between the routing strategy matrix of the current period and the previous period, verify the traffic splitting strategy in combination with the preset smoothing constraint rules, and then execute it. The link status and service transmission results during this round of transmission are stored in the historical dataset.

[0050] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0051] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer software product, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning-based dynamic satellite traffic allocation method for high and low orbit satellites, characterized in that, include: Real-time acquisition of instantaneous basic parameters of high-orbit and low-orbit satellite links, and acquisition of priority labels and real-time traffic requirements of each service data stream currently to be scheduled; Retrieve the historical link state sequence within a preset period, generate high- and low-orbit link state prediction sequences for multiple future decision periods through a lightweight time series prediction model, and concatenate the real-time state parameters with the state prediction sequences to form the input state set of the deep reinforcement learning model. The deep reinforcement learning model dynamically adjusts the priority weights of each service flow based on the electromagnetic interference level of the current link, outputs the traffic allocation ratio of each service flow on the high-speed and low-speed rail links, forms a routing policy matrix, and performs service diversion scheduling based on the routing policy matrix. The routing strategy matrix of the current cycle is compared with that of the previous cycle. The traffic splitting strategy is verified and implemented after combining the preset smoothing constraint rules. The link status and service transmission results of this round of transmission are stored in the historical dataset.

2. The method for dynamic satellite traffic allocation in high and low orbits using deep reinforcement learning according to claim 1, characterized in that, The dynamic adjustment of the priority weights of each service flow includes: In the formula, Let i be the dynamic priority weight of the i-th service flow under the current electromagnetic interference level. The normalized electromagnetic interference level for the current decision-making cycle. Let the base priority weight be the i-th business flow. is the weight adjustment coefficient for the i-th business flow.

3. The method for dynamic satellite traffic splitting in high and low orbits using deep reinforcement learning according to claim 2, characterized in that, It also includes calculating the multi-dimensional QoS utility value of a single service flow, serving as the basic calculation unit for overall utility evaluation: In the formula, Let i be the multidimensional QoS utility value of the i-th service flow. The actual transmission bandwidth obtained by the i-th service flow. The actual round-trip delay obtained by the i-th service flow is Let be the actual bit error rate obtained by the i-th service flow. This represents the bandwidth compliance threshold for the i-th type of service. Let be the latency tolerance threshold for the i-th type of service. The utility coefficient is the one representing bandwidth revenue. The utility coefficient is the dimension of delay loss. This is the utility coefficient for the bit error loss dimension.

4. The method for dynamic diversion of high and low orbit satellites using deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning model is a deep Q-network model, and value iteration and optimal policy inference are performed based on the deep Q-network model. In the formula, Let be the optimal action value for performing action a in state s. This is the current state. For the current action, For the state at the next moment, This is an optional action for the next moment. The immediate reward obtained by performing action a in state s. As a discount factor, For the set of all selectable actions, The Bellman equation is a Q-function.

5. The method for dynamic diversion of high and low orbit satellites using deep reinforcement learning according to claim 1, characterized in that, The priority labels of each business data stream are divided into three levels, including the highest priority corresponding to emergency control instructions, the second highest priority corresponding to video transmission, and the ordinary priority corresponding to sensor telemetry data. Link state parameters are sampled and obtained by the channel awareness module on the terminal side at a fixed decision period, and the sampling period is consistent with the decision period.

6. The method for dynamic satellite traffic allocation in high and low orbits using deep reinforcement learning according to claim 1, characterized in that, The comparison of the differences between the routing strategy matrix of the current period and the previous period includes: Retrieve the routing strategy matrix that was finally implemented in the previous decision cycle, and perform element-wise difference calculation with the initial routing strategy matrix output by the current deep reinforcement learning model. Quantize the overall difference between the two matrices using the norm, and compare the calculated difference value with the preset smoothing threshold. If the difference value does not exceed the threshold, the current strategy is determined to meet the smoothing constraint requirements and is executed; if the difference value exceeds the threshold, the allocation ratio of ordinary priority business and second-highest priority business is compressed successively according to the business priority from low to high.

7. The method for dynamic satellite traffic allocation in high and low orbits using deep reinforcement learning according to claim 6, characterized in that, It also includes the quantitative calculation of instant rewards, including: In the formula, For the current decision-making cycle in state Next action Instant rewards received This represents the current state of the decision-making cycle. This is the routing strategy matrix for the current decision-making cycle. This is the routing strategy matrix from the previous decision cycle. The penalty coefficient for strategy switching. This represents the total QoS utility value of the system under the current state when implementing the current policy.

8. A deep reinforcement learning-based dynamic satellite traffic splitting method for high and low orbit satellites according to claim 6, characterized in that, It also includes writing the complete set of input states corresponding to this round of decision, the verified final routing strategy matrix, the calculated instant reward value, and the actual transmission results such as the actual transmission bandwidth, round-trip delay, bit error rate and actual delivery rate of various services of the high-speed and low-speed links obtained after the end of this decision cycle into the local historical dataset after associating and encapsulating them according to a unified timestamp.

9. A deep reinforcement learning-based dynamic satellite traffic splitting method for high and low orbit satellites according to claim 6, characterized in that, It also includes offline iterative training of the deep Q-network model based on local historical datasets, extracting historical decision samples in chronological order and substituting them into the value iteration process to update the network weights, gradually optimizing the decision accuracy and stability of the model under different electromagnetic interference levels and link time-varying scenarios, keeping the overall decision framework of the model unchanged during the training process, only updating the internal parameters of the network, and replacing the original model after the training converges.

10. A dynamic satellite traffic diversion system for high and low orbits using deep reinforcement learning, characterized in that, A method for dynamic satellite traffic splitting in high and low orbits according to any one of claims 1-9 using deep reinforcement learning includes: The data processing module is configured to collect instantaneous basic parameters of high-orbit and low-orbit satellite links in real time, and simultaneously obtain the priority labels and real-time traffic requirements of each service data stream to be scheduled. It retrieves historical link state sequences within a preset period, generates high-orbit and low-orbit link state prediction sequences for multiple future decision periods through a lightweight time series prediction model, and concatenates the real-time state parameters with the state prediction sequences to form the input state set of the deep reinforcement learning model. The deep reinforcement learning model dynamically adjusts the priority weights of each service stream according to the electromagnetic interference level of the current link, outputs the traffic allocation ratio of each service stream on the high-orbit and low-orbit links, forms a routing strategy matrix, and performs service diversion scheduling according to the routing strategy matrix. The execution judgment module is configured to compare the difference between the routing strategy matrix of the current period and the previous period, verify the traffic splitting strategy in combination with the preset smoothing constraint rules, and then execute it. The link status and service transmission results during this round of transmission are stored in the historical dataset.

Images