Low-orbit satellite cooperative routing and resource scheduling method and system

Abstract

The application discloses a low-orbit satellite cooperative routing and resource scheduling method and system, and the method is as follows: S1, periodically collecting real-time running state information of a low-orbit satellite network; S2, inputting the real-time running state information into a preset fuzzy controller, performing fuzzy processing and fuzzy reasoning, and outputting a comprehensive fuzzy evaluation index of a current inter-satellite link and a dynamic reward weight; S3, constructing a deep reinforcement learning model for joint scheduling and routing decision, taking network topology information and the comprehensive fuzzy evaluation index as inputs of a state space, constructing a reward function based on the dynamic reward weight, and outputting a joint action containing a next-hop routing node and a physical layer resource allocation parameter through a policy network; and S4, executing the joint action in the low-orbit satellite network, collecting environment state transition feedback and an instant reward, and updating network parameters of the deep reinforcement learning model.

Description

Technical Field

[0001] This invention belongs to the field of satellite communication and artificial intelligence technology, specifically relating to a method and system for low-orbit satellite cooperative routing and resource scheduling. Background Technology

[0002] The global low Earth Orbit (LEO) satellite constellation has moved beyond the simple network verification phase and entered a period of large-scale, high-density commercial deployment and technological iteration. This explosive growth in LEO satellite constellations leads to highly dynamic topology changes, resulting in frequent inter-satellite link (ISL) disconnections and limited onboard energy and computing resources. Traditional heuristic routing algorithms (such as OSPF and Dijkstra's algorithm) suffer from extremely high signaling overhead and are prone to routing loops when dealing with highly dynamic satellite network topologies. While conventional reinforcement learning (RL) algorithms can approximate optimal policies, they often suffer from slow convergence, difficulty in joint optimization of multiple objectives (such as throughput, latency, and power consumption), and policy oscillations in continuous action spaces when faced with "fuzzy states" caused by sudden traffic surges or physical link fading.

[0003] To address the shortcomings of existing technologies, this invention proposes a low-Earth orbit (LEO) satellite resource scheduling and routing framework that combines fuzzy control logic (an intelligent control method based on fuzzy mathematics, fuzzy linguistic variables, and fuzzy logic inference) with reinforcement learning algorithms (especially the PPO algorithm). This framework considers the high dynamic characteristics and multi-objective optimization requirements of LEO satellite networks, employs the PPO (Proximity Policy Optimization) algorithm as the foundation for reinforcement learning, and introduces a fuzzy inference system (FIS) to handle the uncertainty of inter-satellite link states, achieving dynamic reward shaping and smooth control of the action space. This invention addresses environmental uncertainty through fuzzy logic and dynamically guides the reward function and exploration mechanism of reinforcement learning, thereby reducing end-to-end latency and onboard energy consumption while ensuring network throughput. Summary of the Invention

[0004] To address the shortcomings of existing heuristic routing algorithms, such as high signaling overhead and susceptibility to loops in highly dynamic satellite network topologies, and the slow convergence, difficulty in multi-objective optimization, and oscillation of continuous action space strategies when conventional reinforcement learning faces fuzzy states, this invention proposes a low-Earth orbit satellite cooperative routing and resource scheduling method and system that integrates fuzzy control and reinforcement learning. This system features state dimensionality reduction, dynamic reward shaping, cross-layer joint decision-making (routing + bandwidth + power), and high topology robustness.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The low-Earth orbit satellite cooperative routing and resource scheduling method includes the following steps:

[0007] Step 1: Network status awareness and data collection.

[0008] Periodically collect real-time operational status information of the low-Earth orbit satellite network, including but not limited to: the available computing resources, queue buffer length, and remaining energy of the current satellite node, as well as the propagation delay, packet loss rate, and available bandwidth of adjacent inter-satellite links.

[0009] Step 2: Link state assessment and dynamic generation of reward weights based on fuzzy control. The link state parameters collected in Step 1 are used as input variables for the fuzzy controller.

[0010] Fuzzification processing: Queuing delay, link load, and packet loss rate are mapped to fuzzy sets (e.g., low, medium, high); wherein, the queue buffer length in the real-time running status information is used to characterize queuing delay, and the available bandwidth is used to characterize link load.

[0011] Fuzzy reasoning: Based on a preset fuzzy rule base (e.g., "If the load is high and the packet loss rate is high, then the link quality is extremely poor"), output the comprehensive fuzzy evaluation index of the current link and the dynamic reward weight (including latency weight, bandwidth weight and energy consumption weight).

[0012] Step 3: Construct a joint scheduling and routing model based on deep reinforcement learning.

[0013] Deploy reinforcement learning agents in the controller of a Software-Defined Satellite Network (SDSN). State space: Contains the source node, destination node, network topology matrix, and the comprehensive fuzzy evaluation index output from step two. Action space: Contains the next-hop routing node selection for data packets, as well as the bandwidth resources and power levels allocated to the traffic flow. Reward function: Constructed based on the dynamic reward weights output by the fuzzy controller, used to guide the agent to balance latency, throughput, and energy consumption.

[0014] Step 4: Model training and action execution.

[0015] The reinforcement learning agent outputs an action based on the current state, the network executes the action and provides feedback on the next state and an immediate reward.

[0016] The innovation of this invention lies in introducing a fuzzy inference system (FIS) as a pre-filter for the PPO model. Through membership functions, complex low-level physical parameters are mapped to high-dimensional semantic features (such as "link congestion" and "node health"). A fuzzy dynamic reward shaping mechanism is proposed. The system utilizes an independent fuzzy controller to evaluate the macroscopic load of the entire constellation and the current lifecycle status of the satellites in real time, dynamically adjusting the weight parameters of each sub-objective in the PPO reward function. A multi-dimensional continuous-discrete hybrid action space is constructed. The PPO agent deployed on the satellite outputs the next-hop forwarding decision (discrete action) while simultaneously outputting the inter-satellite link transmit power and available bandwidth allocation ratio (continuous action), achieving true cross-layer joint optimization. An innovative layer of action secondary smoothing control based on fuzzy confidence intervals is added between the model output and the underlying hardware execution.

[0017] The core technical solution of this invention, based on fuzzy logic, uses the queue length, remaining power of satellite nodes, and physical quality indicators of inter-satellite links (such as signal-to-noise ratio and Doppler shift) as inputs. These are mapped to low-dimensional fuzzy sets of "link congestion" and "node health" through a fuzzy inference system (FIS). In the construction of heterogeneous state and action spaces, reinforcement learning agents are deployed on distributed satellite nodes. The state space S includes the local satellite state, the state of one-hop neighbor nodes, and the fuzzy metrics output by the aforementioned fuzzy inference module. The action space A outputs the next-hop routing decision, satellite transmit power, and bandwidth allocation ratio. In the PPO algorithm based on fuzzy reward shaping, the near-end policy optimization (PPO) algorithm is used for online training of the network scheduling strategy. The fuzzy controller dynamically adjusts the weights of throughput, latency, and energy consumption in the PPO reward function based on the overall load balancing of the current constellation.

[0018] As a preferred approach, in step one, network state awareness and data acquisition:

[0019] The real-time operational status information of the low-Earth orbit (LEO) satellite network is collected periodically. First, the status of the satellite nodes is quantified. During operation, LEO satellites are responsible not only for data packet forwarding but also for on-board edge computing tasks. The physical quantities to be quantified include:

[0020] Available computing resources: Calculated by monitoring CPU clock cycles or computing pool capacity. Let the total computing power of node i at time t be C. total The current set of tasks being processed is K(t), and the available computing resources are... The calculation formula is: (1);

[0021] This represents the computing resources used by the k-th task.

[0022] Queue Buffer Length: Due to the limited storage resources of satellite nodes, the length of the buffer queue directly reflects the congestion level of the nodes. A discrete-time fluid flow model is used for modeling, with a time step of 1. The maximum capacity of the cache in node i is Q. max The queue buffer length Q at time t. i The update process for (t) is as follows: (2);

[0023] Let be the data flow rate arriving at node i at time t. Let t be the data packet service rate (i.e., processing and forwarding rate) of node i at time t.

[0024] Remaining Energy from Solar Panels: Low-Earth orbit satellites rely on solar panels and batteries for power. Due to the periodic alternation of sunlit and shadowed areas, the remaining energy must be dynamically calculated considering the charging and discharging states. Let the remaining energy at node i at time t be... : (3);

[0025] This represents the charging power of the solar panel at time t (this value is 0 when the solar panel is in the Earth's shadow). The total power consumption of the node, including the base power consumption of the onboard payload. Data reception power consumption and data transmission power consumption .

[0026] Inter-satellite link propagation delay: When low-Earth orbit satellites are in high-speed motion, the physical distance between adjacent satellites is time-varying, causing the propagation delay to change dynamically. Let the three-dimensional position vectors of satellite i and satellite j in the geocentric gravitational coordinate system (ECEF) at time t be respectively... and The physical distance of inter-satellite links for: (4);

[0027] This leads to the inter-satellite link propagation delay. : (5);

[0028] c is the speed of light in a vacuum.

[0029] Packet loss rate: Inter-satellite links are affected by cosmic ray interference, Doppler shift, and link disconnections caused by topology changes. Packet loss rate is a core indicator for evaluating link reliability. A sliding time window mechanism is used for statistical analysis, with a statistical time window length of T. w Within this window, the total number of data packets sent by node i to node j is N. tx The number of packets that successfully received acknowledgment frames (ACKs) is N. ack Packet loss rate PLR ij (t) is defined as: (6);

[0030] Available bandwidth: Available bandwidth determines the maximum new traffic flow that the link can currently accommodate. It is limited by the physical channel capacity and the currently allocated bandwidth. Let the total physical channel capacity (maximum bandwidth) of inter-satellite link (i,j) be... At time t, the available bandwidth for: (7);

[0031] Let be the set of all service flows that pass through link (i,j) at time t. This represents the actual bandwidth used by the service flow f.

[0032] To prevent gradient vanishing or weight imbalance when inputting data of different dimensions into step two (fuzzy control) or step three (reinforcement learning neural network), the system maps all state parameters to the [0, 1] interval after completing the initial data acquisition. Taking the remaining energy as an example, the normalized state values... The calculation is as follows: (8);

[0033] This is the maximum battery capacity. The minimum safe power threshold for maintaining basic satellite operation. After normalization, the data matrix containing these six indicators will serve as the observation state vector S. t The data is then formally input into the fuzzy controller and the reinforcement learning agent, S. t for: .

[0034] As a preferred approach, step two involves dynamic generation of link state assessment and reward weights based on fuzzy control. This involves using the link state parameters collected in step one. As input variables to the fuzzy controller, queuing delay, link load, and packet loss rate are mapped to fuzzy sets (e.g., low, medium, high) during fuzzy processing. During fuzzy inference, based on a preset fuzzy rule base, the system outputs a comprehensive fuzzy evaluation index for the current link, along with dynamic reward weights (including delay weight, bandwidth weight, and energy consumption weight).

[0035] Fuzzification aims to map the determined physical quantities (precise values) collected in step one to membership degrees in a fuzzy set (within the range [0,1]), so that the fuzzy logic system can process them. Determining the input variables and domain: Based on the characteristics of the low-Earth orbit satellite network, the following three normalized key states are selected as the input variables for the fuzzy controller: queuing delay. The queue length is derived from step one, and the domain is set to [0,1]. Link load. Defined as The domain is set to [0,1]. Packet loss rate Take the value calculated in step one. The domain is set to [0,1].

[0036] Then, fuzzy sets and membership functions are defined, and three fuzzy linguistic variables are defined for each input variable: Low (L), Medium (M), and High (H). The triangular membership function, which has low computational complexity and is easy to implement in hardware, is used for mapping. Let the input variable be... Trigonometric membership function The general mathematical expression for is defined as follows: (9);

[0037] Where a, b, and c are parameters that determine the shape of the triangular membership function (i.e., the left boundary, vertex, and right boundary). (Based on link load...) For example, the parameter configuration of each fuzzy set can be set as follows: Low (L): a=0, b=0, c=0.4, Medium (M): a=0.2, b=0.5, c=0.8, High (H): a=0.6, b=1.0, c=1.0. Input any precise value, and the system will calculate the membership degree of that value in the three fuzzy sets "Low", "Medium", and "High" through the above functions.

[0038] Then, fuzzy inference is performed. The fuzzy inference stage utilizes a pre-established fuzzy rule base and calculates the output fuzzy set based on the Mamdani inference method. Here, the output variable of the fuzzy rule base is the comprehensive evaluation state (y), and its fuzzy linguistic variables are set to five levels: Very Bad (VB), Bad (B), Normal (N), Good (G), and Excellent (E). The system contains several rules in the form of IF-THEN. The k-th fuzzy rule can be expressed as: (10);

[0039] For example: "IF queuing delay is high (H) AND link load is high (H) AND packet loss rate is high (H) THEN overall evaluation status is very poor (VB)".

[0040] When calculating the trigger strength of a rule and its output membership using the Min-Max algorithm, fuzzy inference is required using a composition operator. First, the trigger strength of the antecedent of the k-th rule is calculated using the intersection (T-norm, where the minimum value operator is used). : (11);

[0041] Then, using the trigger strength For output fuzzy sets The membership function is truncated to obtain the output membership function of a single rule. : (12);

[0042] Then, the outputs of all rules are aggregated using the union (S-norm, here taking the maximum value operator) to obtain the final comprehensive output fuzzy set. : (13);

[0043] In defuzzification and dynamic weight generation, defuzzification is performed to transform the output fuzzy set obtained from fuzzy inference. Transformed into a specific precise value (i.e., the comprehensive fuzzy evaluation index I) eval The centroid of area (CoG) method is used. This method has good smoothness, and the formula is as follows: (14);

[0044] The comprehensive fuzzy evaluation index derived under this formula . A value closer to 1 indicates a higher overall quality of the link; a value closer to 0 indicates link congestion or extremely deteriorated environment. The dynamic reward weight generation matrix is ​​used to obtain... Then, the system maps it to reward weights (delay weights) in a reinforcement learning environment using a preset dynamic transformation matrix. Bandwidth weight Energy consumption weight The dynamic generation of weights satisfies the constraints. Its mathematical transformation function is defined as follows:

[0045] Delay weighting: The worse the link quality (severe congestion), the more time is required to penalize the delay in order to force the route to avoid this node.

[0046] (15);

[0047] Bandwidth weighting: When the link quality is good, maximizing throughput is encouraged.

[0048] (16);

[0049] Energy consumption weight: As a margin balancing term, it can be based on the remaining energy state collected by the node in step one. , and make independent adjustments.

[0050] (17);

[0051] Through the above mathematical process, the fuzzy controller successfully transforms the multidimensional and complex underlying physical link state into evaluation indices and target weights that the reinforcement learning agent can directly read and utilize, thereby driving the update of the Q value in step three.

[0052] As a preferred option, step three is to construct a joint scheduling and routing model based on deep reinforcement learning.

[0053] To deploy reinforcement learning agents in the controller of a software-defined satellite network (SDSN), and to achieve deep coupling between deep reinforcement learning (DRL) and fuzzy control, as well as joint optimization of resource scheduling and routing, a rigorous mathematical model of the problem is first performed using Markov decision process (MDP), then a deep reinforcement learning network architecture is constructed, and finally the algorithm is executed and the model is trained.

[0054] The state space includes the source node, destination node, network topology matrix, and the comprehensive fuzzy evaluation index output from step two. The action space includes the next-hop routing node selection for the data packet, as well as the bandwidth resources and power levels allocated to the traffic flow. The reward function is constructed based on the dynamic reward weights output by the fuzzy controller and is used to guide the agent to balance latency, throughput, and energy consumption.

[0055] (1) Modeling of Markov Decision Process (MDP) for Joint Scheduling and Routing: The dynamic decision-making process of the low-Earth orbit satellite network is modeled as a five-tuple Markov decision process: M={S,A,P,R,γ}.

[0056] First, construct the state space (S). At time t, the state vector perceived by the agent is... Defined as: (18);

[0057] in, The representative service flow information includes the source node number, destination node number, data size, and quality of service (QoS) requirements of the current data packet. Let the network topology matrix reflect the real-time connectivity between N satellites. If there is a link between satellites i and j, then... =1, otherwise 0. It is a comprehensive fuzzy evaluation vector, which is the output of step two. It represents the comprehensive quality index of the K links adjacent to the current node. It provides the agent with "semantic-level" environmental features that go beyond the original physical parameters.

[0058] Reconstruct the Action Space (A) for the actions performed by the agent. It is a mixed-dimensional vector that enables synchronized decision-making for routing and resources: (19);

[0059] ∈{1,…,K} represents the next hop selection, choosing one from the current node's neighboring active satellites as the forwarding node (routing decision). ∈[B min B max This refers to bandwidth allocation, specifically the bandwidth allocated to the selected link for this service flow (resource scheduling decision). ∈[P min ,P max [This refers to the transmission power level, which adjusts the transmission power of the satellite antenna to cope with link loss and balance energy consumption.]

[0060] Finally, a joint reward function (Rt) is constructed, which is crucial for guiding model convergence. This invention utilizes the dynamic weights output in step two to construct an adaptive reward function: (20);

[0061] in: The delay term is the normalized total delay (propagation + queuing) generated by this forwarding. The throughput term represents the normalized data transmission rate that the allocated bandwidth can support. The energy consumption term represents the energy consumed in this transmission. Weight , , It is generated in real time by the fuzzy controller. When network congestion occurs, the fuzzy controller increases... This makes the reward function extremely sensitive to latency, forcing the agent to find low-latency paths.

[0062] (2) Constructing the network architecture: This invention recommends using the Actor-Critic architecture to handle the action space that is a mixture of continuous and discrete.

[0063] Its policy network (Actor Network, π) θ The Actor network is responsible for adjusting the current state. Generate actions Its input layer receives Graph Convolutional Neural Network (GCN) is used to process the topology matrix G. t Finally, a Softmax layer is output, which outputs the probability distribution of selecting each neighboring node, and a Sigmoid / Tanh layer, which outputs the normalized bandwidth and power values.

[0064] Specifically, at time t, the policy network receives the joint state input. Due to the topological matrix For data structures with non-Euclidean origin, ordinary multilayer perceptrons (MLPs) have difficulty extracting spatial dependencies. Therefore, graph convolutional neural networks (GCNs) are used in parallel with MLPs for feature dimensionality reduction and fusion.

[0065] In graph structure feature extraction (GCN branch), the network topology matrix is... As an adjacency matrix A of an undirected graph (containing self-connections, i.e.) Introducing the degree matrix. ,in Spatial connectivity features of all network nodes are extracted through L-layer graph convolution operations. The feature propagation formula for the l-th layer is: (twenty one);

[0066] in, For the initialized node feature matrix, Let h be the trainable weight matrix of the l-th layer. After extraction and flattening, we obtain the graph structure feature vector h. graph In vector feature extraction (MLP branch), business flow requirements are incorporated. The comprehensive fuzzy evaluation vector output from step two The data is then concatenated and input into a multi-layer fully connected network. (twenty two);

[0067] Then, feature fusion is performed, and the outputs of the two branches are concatenated and dimensionality reduced in a fully connected layer to obtain the hidden layer state representation. As a shared input for subsequent action output layers: (twenty three);

[0068] In the discrete action output layer, routing selection (a discrete action) involves choosing the next-hop routing node. The Actor network utilizes a fully connected sublayer to share features. Mapped to a Logits vector with dimension equal to the number K of adjacent active links of the current node. : (twenty four);

[0069] Subsequently, the vector is transformed into a probability distribution for selecting each neighboring node using the Softmax activation function: (25)

[0070] During the model training phase, sampling is performed according to this probability distribution to encourage exploration; during the actual deployment and inference phase, the node with the highest probability value is directly selected as the next hop.

[0071] In the continuous action output layer, bandwidth and power allocation (Resource Allocation) is based on the bandwidth resources and transmit power levels allocated to the traffic flow (which belong to continuous actions). The Actor network uses a smooth activation function to limit the network output within a reasonable physical range.

[0072] Normalize resource output, and calculate bandwidth allocation factor by using a fully connected layer combined with the Sigmoid activation function (mapping the output to the (0,1) interval). and power allocation factor : (26); (27);

[0073] Finally, physical dimension restoration is required. Combining the minimum and maximum physical channel capacity specified in the protocol, the normalization factor is linearly mapped to the actual physical allocation value and sent to the underlying hardware for execution.

[0074] Actual allocated bandwidth: (28);

[0075] Actual transmission power: (29);

[0076] In summary, the policy network In a single forward computation, joint actions are output synchronously using deterministic formulas or probability distributions. This enables joint optimization across layers (network layer routing and physical / MAC layer resources).

[0077] (3) Specific algorithm execution and training process, experience collection, and running the current strategy using the SDSN (Software Defined Satellite Network) controller. , will the historical trajectory {s t a t , R t ,s t+1 Store it in the Experience ReplayBuffer.

[0078] Then, gradient updates are performed under fuzzy guidance, using data drawn from the replay pool to update the weight parameters of the policy network (Actor) and value network (Critic). A mini-batch of samples of size M is randomly drawn from the empirical replay pool. , Refers to the historical trajectory in the playback pool, value network (Critic, In the update of ), the value network is used to evaluate a certain state s. t The long-term expected return. First, calculate the target value. : (30);

[0079] in, This is the discount factor. The optimization objective of the Critic network is to minimize the mean squared error (MSE Loss) between the predicted and target values, and its loss function is... and its parameters The gradient descent update formula is: (31); (32);

[0080] in is the learning rate of the Critic network.

[0081] Randomly draw samples from the pool and calculate the advantage function. (33);

[0082] Update the Actor parameters θ using the policy gradient algorithm: (34);

[0083] To encourage exploration, a policy entropy term is added to the objective function. ,That The learning rate of the Actor network. is the entropy regularization coefficient.

[0084] (4) Online decision-making and distribution: In actual operation, the controller inputs... (Including the latest fuzzy evaluation index), the Actor network outputs the optimal action. .

[0085] Once the model has converged after offline or semi-online training in the SDSN controller, it will enter the real-time deployment and execution phase. Real-time state input (Forward Pass) is generated when real traffic arrives; the controller collects network topology and fuzzy evaluation index in real time to form the current state. Input into the already trained Actor network Deterministic execution, unlike probabilistic sampling during training, employs a greedy policy to output the optimal action during online deployment to ensure network performance stability. In discrete routing decisions, the node with the highest Softmax output probability is selected. .

[0086] In continuous resource decision-making, the deterministic expected value of the network's forward computation is directly output, i.e., the bandwidth. With power In the Southbound Interface Dispatch, the SDSN controller utilizes the satellite-to-ground / inter-satellite link control channel and, through the extended OpenFlow or P4 protocol (SDN Southbound protocol), parses actions into underlying commands and issues them. During routing table updates, flow table entries are sent to the source node and LEO satellites along the route, specifying the destination IP segment to match. In resource scheduling configuration, control frames are sent to the MAC layer scheduler and physical layer radio frequency unit to lock the link guarantee bandwidth. And set the transmitter gain to match the power level. In this way, the controller sends routing instructions (next hop) and resource instructions (bandwidth / power) to the corresponding low-Earth orbit satellite nodes via OpenFlow or a similar protocol.

[0087] This completes the first successful implementation of the semantic output I of the fuzzy controller. eval By integrating with the original state space of reinforcement learning, the problem of extracting state features in dynamic LEO environments is solved. Simultaneously, the reward function is no longer a fixed percentage but a "scalable reward" that dynamically changes with fuzzy rules, greatly improving the model's obstacle avoidance capability during peak traffic periods. Joint optimization of routing hop count, spectrum utilization, and satellite energy consumption is achieved, rather than a single-dimensional greedy pathfinding approach.

[0088] As a preferred option, step four is model training and action execution.

[0089] The reinforcement learning agent deployed in the SDSN controller here achieves closed-loop iteration and eventually converges through continuous interaction with the low-Earth orbit satellite network environment, utilizing the temporal difference (TD) algorithm and policy gradient mechanism.

[0090] (1) Action Execution and State Transition

[0091] At time t, the agent is based on the policy network Output joint action After the underlying low-Earth orbit satellite network performs this action, the network environment changes. The state transition process of the environment follows Markov properties and can be expressed by the state transition probability function P: Simultaneously, the environment (in conjunction with the fuzzy controller from step two) evaluates the physical feedback generated by the action (such as new latency, throughput, and energy consumption changes) and generates an immediate reward. This interactive process generates single-step experience data. The data are then stored in the experience replay pool D for subsequent offline training.

[0092] (2) Time Difference (TD) Target and TD Error Calculation: This is used to evaluate the selected action. In state To assess the advantages and disadvantages of different approaches, a time-difference learning method based on the Bellman Equation is introduced. The agent utilizes a Critic Network... To approximate the state value function. For any empirical tuple drawn from the replay pool D. Its Time Difference Target (TD Target) Defined as the sum of the immediate reward and the value of the next state discount: (35);

[0093] in [0,1) represents the discount factor, used to balance the weight of current rewards versus future rewards. Based on this, the Time Difference Error (TD Error) is calculated. This is used to quantify the deviation between the current value network's predicted values ​​and the actual target. (36);

[0094] TD error Essentially, it is the advantage function A(s) i , a i The unbiased estimate of ) will serve as a benchmark for subsequent parameter updates of the Actor and Critic networks.

[0095] (3) Derivation of Gradient Update for Dual Networks

[0096] The model training alternately updates the Critic and Actor networks by minimizing the loss function and maximizing the expected total reward, respectively.

[0097] Value Network (Critic, with parameters) In the update of the value network, the optimization objective is to minimize the mean squared TD error. A mini-batch mean squared error loss function with M samples is defined. (37);

[0098] Update parameters using gradient descent (e.g., the Adam optimizer). (38);

[0099] in, is the learning rate of the Critic network.

[0100] Policy network (Actor, with parameters) In the update of the policy network, the goal is to find the optimal policy. This maximizes the expected cumulative reward from the initial state. Using the policy gradient theorem, the policy objective function... The gradient can be expressed as: (39);

[0101] This indicates the expectation value. To prevent the strategy from getting trapped in local optima too early, an information entropy regularization term is introduced. Actor parameters The gradient ascent update formula is: (40);

[0102] in The learning rate of the Actor network. The entropy coefficient is used to control the degree of exploration.

[0103] (4) Soft Update Mechanism of the Target Network

[0104] Because in calculating the TD target y i It directly used its own network. This can cause the target to move continuously, easily leading to training oscillations (non-convergence). This invention introduces a target network mechanism in step four. A target network with the exact same structure as the online Actor and Critic is set up. and After each iteration or several cycles, a soft update formula is used to smoothly track the parameters of the target network: (41); (42);

[0105] Among them, soft update coefficient Setting it to 0.005 significantly improves the learning stability of reinforcement learning when dealing with dynamic topology changes in low-Earth orbit satellites. When the time difference error... When the value approaches 0 or reaches the preset training epoch threshold, the model training is complete, and the parameters are fixed for online execution.

[0106] The reinforcement learning agent then outputs an action based on its current state, the network executes the action, and provides feedback on the next state and an immediate reward.

[0107] This invention also discloses a low-Earth orbit satellite cooperative routing and resource scheduling system for performing the above method, which includes the following modules:

[0108] The status awareness module is used to periodically collect real-time operational status information of the low-Earth orbit satellite network and perform normalization processing.

[0109] The fuzzy evaluation module is used to perform fuzzy reasoning on the state information and output a comprehensive fuzzy evaluation index and dynamic reward weight.

[0110] The deep reinforcement learning decision module is used to output joint routing and resource scheduling actions based on the network topology and the comprehensive fuzzy evaluation index, and the reward function constructed based on the dynamic reward weight.

[0111] The execution and closed-loop training module is used to issue flow tables and resource configuration instructions to the underlying satellite nodes, collect environmental feedback, and update the reinforcement learning network parameters.

[0112] Compared with the prior art, the beneficial effects of this invention are:

[0113] This invention provides a method and system for cooperative routing and resource scheduling of low-Earth orbit (LEO) satellites based on reinforcement learning and fuzzy control. Existing deep reinforcement learning (DRL) routing algorithms are prone to the curse of dimensionality when dealing with large and rapidly changing LEO satellite constellations, leading to difficulty in training convergence. Traditional reinforcement learning routing models typically use fixed-weight reward functions, failing to consider the different needs of network idle and congested periods. To address these shortcomings, this invention innovatively introduces a fuzzy controller as a pre-evaluation module, reducing and fusing multi-dimensional continuous underlying physical indicators (queueing delay, link load, packet loss rate) into a semantic-level "comprehensive fuzzy evaluation index." This mechanism significantly compresses the state space complexity of the reinforcement learning agent, increasing the convergence speed of the model in complex constellation networks by several times and reducing the computational overhead of the controller. Existing technologies typically separate data routing at the network layer from resource allocation at the physical / MAC layer, easily leading to local suboptimal solutions. A dynamic adaptive reward mechanism is constructed to enhance resistance to sudden congestion. Utilizing the reasoning capabilities of fuzzy logic, reward weights are dynamically generated based on the real-time degradation of local links (e.g., automatically and significantly increasing the delay penalty weight during congestion). This "scalable reward" mechanism endows the agent with strong environmental adaptability, enabling millisecond-level intelligent detours and load balancing in sudden high-traffic scenarios, effectively reducing end-to-end latency and data packet loss rate across the entire network. It achieves cross-layer joint optimization, maximizing satellite node resource utilization and constructing a heterogeneous hybrid action space. This allows the policy network to simultaneously output discrete next-hop routing decisions and continuous bandwidth and power allocation decisions within a forward propagation cycle. This cross-layer joint scheduling mechanism not only improves the spectrum utilization of inter-satellite links but also effectively balances the energy consumption of each satellite node through power-level control, extending the on-orbit service life of battery-constrained low-Earth orbit satellites. It enhances the stability and robustness of policies under highly dynamic topologies by integrating Graph Convolutional Neural Networks (GCNs) into the underlying design of the policy network, enabling real-time capture and extraction of spatial features of the global topology; simultaneously, a soft update mechanism for the target network is introduced during the training phase. This combination makes the routing policy less prone to training oscillations when facing drastic topology changes caused by high-speed satellite movement, ensuring the smoothness and stability of online routing command issuance. Attached Figure Description

[0114] Figure 1 This is a flowchart of a satellite cooperative routing and resource scheduling method based on reinforcement learning and fuzzy control according to an embodiment of the present invention; Figure 2 A simulation flowchart of a satellite cooperative routing and resource scheduling method based on reinforcement learning and fuzzy control; Figure 3 This is a schematic diagram illustrating the typical characteristics of a value network loss curve. Detailed Implementation

[0115] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0116] This invention presents a low-Earth orbit (LEO) satellite resource scheduling and routing method based on reinforcement learning and fuzzy control. Through data acquisition and state awareness (environmental assessment), the SDSN controller acts as the "brain" of the entire satellite network, periodically collecting real-time physical operational status data from the underlying LEO satellite nodes. The collected data includes satellite node computing resources, buffer queue length, remaining power, and propagation delay, packet loss rate, and available bandwidth of adjacent inter-satellite links. This multi-dimensional raw physical data is normalized to provide a standard input basis for subsequent intelligent algorithms. Fuzzy control evaluation and reward reshaping (experience-based dimensionality reduction) are then performed. When faced with complex, variable, and noisy physical data, the system uses a "fuzzy controller" to simulate human expert experience for evaluation. First, state dimensionality reduction is performed by inputting queuing delay, link load, and packet loss rate into the fuzzy system. Reasoning is then performed using a pre-set fuzzy rule base (IF-THEN), outputting an intuitive comprehensive fuzzy evaluation index (reflecting the current link's condition). Based on this evaluation index, the reward weights of reinforcement learning are adjusted in real time. In DRL joint scheduling and routing decision-making (brain-based decision-making), after obtaining the network topology and fuzzy evaluation, the reinforcement learning agent (Actor network) within the controller begins cross-layer joint decision-making. First, feature extraction is performed: a Graph Convolutional Network (GCN) is used to process the complex satellite connectivity topology, while a Multilayer Perceptron (MLP) is used to process service flow requirements and fuzzy evaluation indices. Then, actions are output synchronously. In a forward computation process, the agent simultaneously outputs decisions at two levels: the network layer (discrete) (which satellite to select as the next-hop routing node for the data packet (based on Softmax probability)) and the physical layer (continuous) (how much bandwidth and transmission power to allocate to this service flow (based on the Sigmoid / Tanh function)). Finally, action issuance and closed-loop learning (execution and evolution) are performed. After decision generation, the system enters the final stage of execution and self-optimization, issuing instructions. During the distribution process, the SDSN controller translates the routing table and resource configuration parameters into underlying protocol instructions and distributes them to the corresponding satellite nodes for execution, completing the packet forwarding. During feedback and learning, the environmental state transitions. The agent calculates the "instant reward" for this action based on the dynamic weights given by the fuzzy controller in "Step Two." The system stores these experiences in a playback pool and continuously updates the parameters of the Actor and Critic neural networks using the gradient descent algorithm by calculating the time difference (TD) error. As the model iterates and converges, the system gains the ability to adaptively output optimal routing and resource scheduling schemes within milliseconds when facing sudden traffic surges and drastic topology changes.

[0117] like Figure 1 and Figure 2As shown, this embodiment provides a low-Earth orbit satellite resource scheduling and routing method based on reinforcement learning and fuzzy control. The SDSN controller senses the underlying physical data, the fuzzy system evaluates it and dynamically adjusts the optimization target, and the reinforcement learning brain simultaneously makes optimal decisions on "who to send to (routing)" and "how many resources to allocate (scheduling)". Finally, the method is deployed and executed, and continuously evolves based on feedback. The specific steps include:

[0118] Step 1: Network status awareness and data collection.

[0119] Step 2: Link state assessment and dynamic generation of reward weights based on fuzzy control.

[0120] Step 3: Construct a joint scheduling and routing model based on deep reinforcement learning.

[0121] Step 4: Model training and action execution.

[0122] The following describes in detail each step of a low-Earth orbit satellite resource scheduling and routing method based on reinforcement learning and fuzzy control in this embodiment.

[0123] Step 1: Network Status Awareness and Data Acquisition

[0124] The real-time operational status information of the low-Earth orbit satellite network is collected periodically, and the status of the satellite nodes is first quantified:

[0125] (1) Available computing resources Low-Earth orbit satellites, during operation, are responsible not only for data packet forwarding but also for on-board edge computing tasks. Their available computing resources are calculated by monitoring CPU clock cycles or the remaining computing power pool. Let the total computing power of node i at time t be C. total The current set of tasks being processed is K(t), and the available computing resources are... The calculation formula is: ;

[0126] This represents the computing resources used by the k-th task.

[0127] (2) Queue buffer length Due to the limited storage resources of satellite nodes, the length of the cache queue directly reflects the congestion level of the nodes. A discrete-time fluid flow model is used for modeling, with a time step of 1. The maximum capacity of the cache in node i is Q. max The queue buffer length Q at time t i The update process for (t) is as follows: ;

[0128] Let be the data flow rate reaching node i at time t. Let t be the data packet service rate (i.e., processing and forwarding rate) of node i at time t.

[0129] (3) Remaining energy of solar panels Low-Earth orbit satellites rely on solar panels and batteries for power, and there is a periodic alternation between illuminated and shadowed areas. The remaining energy must be dynamically calculated based on the charging and discharging states. Let the remaining energy of node i at time t be... : ;

[0130] This represents the charging power of the solar panel at time t (this value is 0 when the solar panel is in the Earth's shadow). The total power consumption of the node, including the base power consumption of the onboard payload. Data reception power consumption and data transmission power consumption .

[0131] (4) Inter-satellite link propagation delay When low-Earth orbit satellites are in high-speed motion, the physical distance between adjacent satellites is time-varying, causing the propagation delay to change dynamically. Let the three-dimensional position vectors of satellite i and satellite j in the geocentric gravitational coordinate system (ECEF) at time t be respectively... and Physical distance of inter-satellite links for: ;

[0132] This leads to the inter-satellite link propagation delay. : ;

[0133] c is the speed of light in a vacuum.

[0134] (5) Packet loss rate PLR ij (t): Inter-satellite links are affected by cosmic ray interference, Doppler shift, and link disconnections caused by topology changes. Packet loss rate is the core indicator for evaluating link reliability. A sliding time window mechanism is used for statistics, with the statistical time window length set to T. w Within this window, the total number of data packets sent by node i to node j is N. tx The number of packets that successfully received acknowledgment frames (ACKs) is N. ack Packet loss rate PLR ij (t) is defined as: ;

[0135] (6) Available bandwidth The available bandwidth determines the maximum new traffic flow that the link can currently accommodate. It is limited by the physical channel capacity and the currently allocated bandwidth. Let the total physical channel capacity (maximum bandwidth) of inter-satellite link (i,j) be... At time t, the available bandwidth for: ;

[0136] Let be the set of all service flows that pass through link (i,j) at time t. This represents the actual bandwidth used by the service flow f.

[0137] To prevent gradient vanishing or weight imbalance when inputting data of different dimensions into step two (fuzzy control) or step three (reinforcement learning neural network), the system maps all state parameters to the [0,1] interval after completing the initial data acquisition. Taking the remaining energy as an example, the normalized state values... The calculation is as follows: ;

[0138] This is the maximum battery capacity. The minimum safe power threshold for maintaining basic satellite operation. After normalization, the data matrix containing these six indicators will serve as the observation state vector S. t The data is then formally input into the fuzzy controller and the reinforcement learning agent, S. t for: .

[0139] Step 2: Link state assessment and dynamic generation of reward weights based on fuzzy control.

[0140] The link status parameters collected in step one As input variables to the fuzzy controller, queuing delay, link load, and packet loss rate are mapped to fuzzy sets (e.g., low, medium, high) during fuzzy processing. During fuzzy inference, based on a preset fuzzy rule base, the system outputs a comprehensive fuzzy evaluation index for the current link, along with dynamic reward weights (including delay weight, bandwidth weight, and energy consumption weight).

[0141] Fuzzification aims to map the determined physical quantities (precise values) collected in step one to membership degrees in a fuzzy set (within the range [0,1]) so that the fuzzy logic system can process them. Determining the input variables and universe of discourse: Based on the characteristics of the low-Earth orbit satellite network, the following three normalized key states are selected as the input variables for the fuzzy controller: queuing delay. The queue length is derived from step one, and the domain is defined as [0, 1]. Link load. Defined as The universe of discourse is set to [0,1]. Packet loss rate Take the value calculated in step one. The domain is set to [0,1].

[0142] (1) Define fuzzy sets and membership functions: Define three fuzzy linguistic variables for each input variable: Low (L), Medium (M), and High (H). The triangular membership function, which has low computational complexity and is easy to implement in hardware, is used for mapping. Let the input variable be... Trigonometric membership function The general mathematical expression for is defined as follows: ;

[0143] Where a, b, and c are parameters that determine the shape of the triangular membership function (i.e., the left boundary, vertex, and right boundary). (Based on link load...) For example, the parameter configuration of each fuzzy set can be set as follows: Low (L): a=0, b=0, c=0.4, Medium (M): a=0.2, b=0.5, c=0.8, High (H): a=0.6, b=1.0, c=1.0. Input any precise value, and the system will calculate the membership degree of that value in the three fuzzy sets "Low", "Medium", and "High" through the above functions.

[0144] (2) Fuzzy Inference: The fuzzy inference stage utilizes a pre-established fuzzy rule base to calculate the output fuzzy set based on the Mamdani inference method. Here, the output variable of the fuzzy rule base is the comprehensive evaluation state (y), and its fuzzy linguistic variables are set to five levels: Very Bad (VB), Bad (B), Normal (N), Good (G), and Excellent (E). The system contains several rules in the form of IF-THEN. The k-th fuzzy rule can be expressed as: ;

[0145] For example: "IF queuing delay is high (H) AND link load is high (H) AND packet loss rate is high (H) THEN overall evaluation status is very poor (VB)".

[0146] When calculating the trigger strength of a rule and its output membership using the Min-Max algorithm, fuzzy inference is required using a composition operator. First, the trigger strength of the antecedent of the k-th rule is calculated using the intersection (T-norm, where the minimum value operator is used). : ;

[0147] Then, using the trigger strength For output fuzzy sets The membership function is truncated to obtain the output membership function of a single rule. : ;

[0148] Then, the outputs of all rules are aggregated using the union (S-norm, here taking the maximum value operator) to obtain the final comprehensive output fuzzy set. : );

[0149] (3) Defuzzification: In defuzzification and dynamic weight generation, the defuzzification process is to transform the output fuzzy set obtained by fuzzy inference into a fuzzy set. Transformed into a specific precise value (i.e., the comprehensive fuzzy evaluation index I) eval The centroid method (CoG) is used. This method has good smoothness, and the formula is as follows: ;

[0150] The comprehensive fuzzy evaluation index derived from this formula . A value closer to 1 indicates a higher overall quality of the link; a value closer to 0 indicates link congestion or extremely deteriorated environment. The dynamic reward weight generation matrix is ​​used to obtain... Then, the system maps it to reward weights (delay weights) in a reinforcement learning environment using a preset dynamic transformation matrix. Bandwidth weight Energy consumption weight The dynamic generation of weights satisfies the constraints. Its mathematical transformation function is defined as follows:

[0151] Delay weighting: The worse the link quality (severe congestion), the more time is required to penalize the delay in order to force the route to avoid this node.

[0152] ;

[0153] Bandwidth weighting: When the link quality is good, maximizing throughput is encouraged.

[0154] ;

[0155] Energy consumption weight: As a margin balancing term, it can be based on the remaining energy state collected by the node in step one. Make independent adjustments.

[0156] ;

[0157] Through the above mathematical process, the fuzzy controller successfully transforms the multidimensional and complex underlying physical link state into evaluation indices and target weights that the reinforcement learning agent can directly read and utilize, thereby driving the update of the Q value in step three.

[0158] Step 3: Construct a joint scheduling and routing model based on deep reinforcement learning

[0159] To deploy reinforcement learning agents in the controller of a software-defined satellite network (SDSN), and to achieve deep coupling between deep reinforcement learning (DRL) and fuzzy control, as well as joint optimization of resource scheduling and routing, a rigorous mathematical model of the problem is first performed using Markov decision process (MDP), then a deep reinforcement learning network architecture is constructed, and finally the algorithm is executed and the model is trained.

[0160] The state space includes the source node, destination node, network topology matrix, and the comprehensive fuzzy evaluation index output from step two. The action space includes the next-hop routing node selection for the data packet, as well as the bandwidth resources and power levels allocated to the traffic flow. The reward function is constructed based on the dynamic reward weights output by the fuzzy controller and is used to guide the agent to balance latency, throughput, and energy consumption.

[0161] (1) Modeling of Markov Decision Process (MDP) for Joint Scheduling and Routing: The dynamic decision-making process of the low-Earth orbit satellite network is modeled as a five-tuple Markov decision process: M={S,A,P,R,γ}.

[0162] First, construct the state space (S). At time t, the state vector perceived by the agent is... Defined as:

[0163] in, The representative service flow information includes the source node number, destination node number, data size, and quality of service (QoS) requirements of the current data packet. The network topology matrix reflects the real-time connectivity between N satellites. If a link exists between satellites i and j, then... =1, otherwise 0. It is a comprehensive fuzzy evaluation vector, which is the output of step two. It represents the comprehensive quality index of the K links adjacent to the current node. It provides the agent with "semantic-level" environmental features that go beyond the original physical parameters.

[0164] Reconstruct the Action Space (A) for the actions performed by the agent. It is a mixed-dimensional vector that enables synchronized decision-making for routing and resources: ;

[0165] ∈{1,…,K} represents the next hop selection, choosing one from the current node's neighboring active satellites as the forwarding node (routing decision). ∈[B min B max This refers to bandwidth allocation, specifically the bandwidth allocated to the selected link for this service flow (resource scheduling decision). ∈[P min , P max [This refers to the transmission power level, which adjusts the transmission power of the satellite antenna to cope with link loss and balance energy consumption.]

[0166] Finally, a joint reward function (R) is constructed. t The reward function is crucial for guiding model convergence. This invention utilizes the dynamic weights output in step two to construct an adaptive reward function: ;

[0167] in: The delay term is the normalized total delay (propagation + queuing) generated by this forwarding. The throughput term represents the normalized data transmission rate that the allocated bandwidth can support. The energy consumption term represents the energy consumed in this transmission. Weight , , It is generated in real time by the fuzzy controller. When network congestion occurs, the fuzzy controller increases... This makes the reward function extremely sensitive to latency, forcing the agent to find low-latency paths.

[0168] (2) Constructing the network architecture: This invention recommends using the Actor-Critic architecture to handle the action space that is a mixture of continuous and discrete.

[0169] Its policy network (Actor Network, π) θ The Actor network is responsible for adjusting the current state. Generate actions Its input layer receives Graph Convolutional Neural Network (GCN) is used to process the topology matrix G. t Finally, a Softmax layer is output, which outputs the probability distribution of selecting each neighboring node, and a Sigmoid / Tanh layer, which outputs the normalized bandwidth and power values.

[0170] Specifically, at time t, the policy network receives the joint state input. Due to the topological matrix For data structures with non-Euclidean characteristics, ordinary multilayer perceptrons (MLPs) have difficulty extracting spatial dependencies. Therefore, graph convolutional neural networks (GCNs) are used in parallel with MLPs for feature dimensionality reduction and fusion.

[0171] In graph structure feature extraction (GCN branch), the network topology matrix is... As an adjacency matrix A of an undirected graph (containing self-connections, i.e.) Introducing the degree matrix. ,in Spatial connectivity features of all network nodes are extracted through L-layer graph convolution operations. The feature propagation formula for the l-th layer is: ;

[0172] in, For the initialized node feature matrix, Let h be the trainable weight matrix of the l-th layer. After extraction and flattening, we obtain the graph structure feature vector h. graph In vector feature extraction (MLP branch), business flow requirements are incorporated. The comprehensive fuzzy evaluation vector output from step two The data is then concatenated and input into a multi-layer fully connected network. ;

[0173] Then, feature fusion is performed, and the outputs of the two branches are concatenated and dimensionality reduced in a fully connected layer to obtain the hidden layer state representation. As a shared input for subsequent action output layers: ;

[0174] In the discrete action output layer, routing selection (a discrete action) involves choosing the next-hop routing node. The Actor network utilizes a fully connected sublayer to share features. Mapped to a Logits vector with dimension equal to the number K of adjacent active links of the current node. : ;

[0175] Then, the vector is transformed into a probability distribution for selecting each neighboring node using the Softmax activation function: ;

[0176] During the model training phase, sampling is performed according to this probability distribution to encourage exploration; during the actual deployment and inference phase, the node with the highest probability value is directly selected as the next hop.

[0177] In the continuous action output layer, bandwidth and power allocation (Resource Allocation) is based on the bandwidth resources and transmit power levels allocated to the traffic flow (which belong to continuous actions). The Actor network uses a smooth activation function to limit the network output within a reasonable physical range.

[0178] Normalize the resource output and calculate the bandwidth allocation factor using a fully connected layer combined with the Sigmoid activation function (mapping the output to the (0, 1) interval). and power allocation factor : ; ;

[0179] (3) Physical Dimension Restoration: Finally, physical dimension restoration is required. Combining the minimum and maximum physical channel capacity specified in the protocol, the normalization factor is linearly mapped to the actual physical allocation value and sent to the underlying hardware for execution.

[0180] Actual allocated bandwidth: ;

[0181] Actual transmission power: ;

[0182] In summary, the policy network In a single forward computation, joint actions are output synchronously using deterministic formulas or probability distributions. This enables joint optimization across layers (network layer routing and physical / MAC layer resources).

[0183] (4) Specific algorithm execution and training process, experience collection, and running the current strategy using the SDSN (Software Defined Satellite Network) controller. , will the historical trajectory {s t a t , Rt ,s t+1 Store it in the Experience ReplayBuffer.

[0184] Then, gradient updates are performed under fuzzy guidance, using data drawn from the replay pool to update the weight parameters of the policy network (Actor) and value network (Critic). A mini-batch of samples of size M is randomly drawn from the empirical replay pool. , Refers to the historical trajectory in the playback pool, value network (Critic, In the update of ), the value network is used to evaluate a certain state s. t The long-term expected return. First, calculate the target value. : ;

[0185] in, This is the discount factor. The optimization objective of the Critic network is to minimize the mean squared error (MSE Loss) between the predicted and target values, and its loss function is... and its parameters The gradient descent update formula is: ); ;

[0186] in is the learning rate of the Critic network.

[0187] Randomly draw samples from the pool and calculate the advantage function: ;

[0188] Update the Actor parameters θ using the policy gradient algorithm: ;

[0189] To encourage exploration, a policy entropy term is added to the objective function. ,That The learning rate of the Actor network. is the entropy regularization coefficient.

[0190] (4) Online decision-making and distribution: In actual operation, the controller inputs s t (Including the latest fuzzy evaluation index), the Actor network outputs the optimal action a. t .

[0191] Once the model has converged after offline or semi-online training in the SDSN controller, it will enter the real-time deployment and execution phase. Real-time state input (Forward Pass) is generated when real traffic arrives; the controller collects network topology and fuzzy evaluation index in real time to form the current state. Input into the already trained Actor network Deterministic execution, unlike probabilistic sampling during training, employs a greedy policy to output the optimal action during online deployment to ensure network performance stability. In discrete routing decisions, the node with the highest Softmax output probability is selected. .

[0192] In continuous resource decision-making, the deterministic expected value of the network's forward computation is directly output, i.e., the bandwidth. With power In the Southbound Interface Dispatch, the SDSN controller utilizes the satellite-to-ground / inter-satellite link control channel and, through the extended OpenFlow or P4 protocol (SDN Southbound protocol), parses actions into underlying commands and issues them. During routing table updates, flow table entries are sent to the source node and LEO satellites along the route, specifying the destination IP segment to match. In resource scheduling configuration, control frames are sent to the MAC layer scheduler and physical layer radio frequency unit to lock the link guarantee bandwidth. And set the transmitter gain to match the power level. In this way, the controller sends routing instructions (next hop) and resource instructions (bandwidth / power) to the corresponding low-Earth orbit satellite nodes via OpenFlow or a similar protocol.

[0193] This completes the first successful implementation of the semantic output I of the fuzzy controller. eval By integrating with the original state space of reinforcement learning, the problem of extracting state features in dynamic LEO environments is solved. Simultaneously, the reward function is no longer a fixed percentage but a "scalable reward" that dynamically changes with fuzzy rules, greatly improving the model's obstacle avoidance capability during peak traffic periods. Joint optimization of routing hop count, spectrum utilization, and satellite energy consumption is achieved, rather than a single-dimensional greedy pathfinding approach.

[0194] Step four: Model training and action execution.

[0195] The reinforcement learning agent deployed in the SDSN controller here achieves closed-loop iteration and eventually converges through continuous interaction with the low-Earth orbit satellite network environment, utilizing the temporal difference (TD) algorithm and policy gradient mechanism.

[0196] (1) Action Execution and State Transition

[0197] At time t, the agent is based on the policy network Output joint action After the underlying low-Earth orbit satellite network performs this action, the network environment changes. The state transition process of the environment follows Markov properties and can be expressed by the state transition probability function P: Simultaneously, the environment (in conjunction with the fuzzy controller from step two) evaluates the physical feedback generated by the action (such as new latency, throughput, and energy consumption changes) and generates an immediate reward. This interactive process generates single-step experience data. The data are then stored in the experience replay pool D for subsequent offline training.

[0198] (2) Time Difference (TD) Target and TD Error Calculation: This is used to evaluate the selected action. In state To assess the advantages and disadvantages of different approaches, a time-difference learning method based on the Bellman Equation is introduced. The agent utilizes a Critic Network... To approximate the state value function. For any empirical tuple drawn from the replay pool D. Its Time Difference Target (TD Target) Defined as the sum of the immediate reward and the value of the next state discount: ;

[0199] in [0, 1) represents the discount factor, used to balance the weight of current rewards versus future rewards. Based on this, the Time Difference Error (TD Error) is calculated. This is used to quantify the deviation between the current value network's predicted values ​​and the actual target. ;

[0200] TD error Essentially, it is the advantage function A(s) i , a iThe unbiased estimate of ) will serve as a benchmark for subsequent parameter updates of the Actor and Critic networks.

[0201] (3) Derivation of Gradient Update for Dual Networks

[0202] The model training alternately updates the Critic and Actor networks by minimizing the loss function and maximizing the expected total reward, respectively.

[0203] Value network (Critic, with parameters) In the update of the value network, the optimization objective is to minimize the mean squared TD error. A mini-batch mean squared error loss function with M samples is defined. ;

[0204] Update parameters using gradient descent (e.g., the Adam optimizer): ;

[0205] in, is the learning rate of the Critic network.

[0206] Policy network (Actor, with parameters) In the update of the policy network, the goal is to find the optimal policy. This maximizes the expected cumulative reward from the initial state. Using the policy gradient theorem, the policy objective function... The gradient can be expressed as: ;

[0207] This indicates the expectation value. To prevent the strategy from getting trapped in local optima too early, an information entropy regularization term is introduced. Actor parameters The gradient ascent update formula is: ;

[0208] in The learning rate of the Actor network. The entropy coefficient is used to control the degree of exploration.

[0209] (4) Soft update mechanism of the target network

[0210] Because in calculating the TD target y i It directly used its own network. This can cause the target to move continuously, easily leading to training oscillations (non-convergence). This invention introduces a target network mechanism in step four. A target network with the exact same structure as the online Actor and Critic is set up. and After each iteration or several cycles, a soft update formula is used to smoothly track the parameters of the target network: ; ;

[0211] Among them, soft update coefficient Setting it to 0.005 significantly improves the learning stability of reinforcement learning when dealing with dynamic topology changes in low-Earth orbit satellites. When the time difference error... When the value approaches 0 or reaches the preset training epoch threshold, the model training is complete, and the parameters are fixed for online execution.

[0212] The reinforcement learning agent then outputs an action based on its current state, the network executes the action, and provides feedback on the next state and an immediate reward.

[0213] In addition, an embodiment of a low-Earth orbit (LEO) satellite network resource scheduling and routing system based on reinforcement learning and fuzzy control is deployed in a software-defined satellite network (SDSN) controller. This system includes: a state awareness module for periodically collecting and normalizing real-time operational status information of the LEO satellite network; a fuzzy evaluation module for performing fuzzy inference on the state information and outputting a comprehensive fuzzy evaluation index and dynamic reward weights; a deep reinforcement learning decision module for outputting joint routing and resource scheduling actions based on a reward function constructed using the dynamic reward weights and the network topology and the comprehensive fuzzy evaluation index; and an execution and closed-loop training module for issuing flow tables and resource configuration instructions to the underlying satellite nodes, collecting environmental feedback, and updating the reinforcement learning network parameters. This achieves the technical objective of a LEO satellite cooperative routing and resource scheduling method based on reinforcement learning and fuzzy control.

[0214] Figure 3This reflects the typical characteristics of the value network loss curve, with the horizontal axis representing the number of simulation iterations and the vertical axis representing the loss function. During training, the reinforcement learning "dataset" is dynamically generated and continuously updated; this is called the Experience Replay Buffer. Before actual deployment, this dataset is generated through interaction between the SDSN controller and the "network simulator." The satellite network simulation platform used in the simulation is NS-3 / OMNeT++, which simulates the physical characteristics of the underlying network, such as packet forwarding, queuing delay, and packet loss rate. STK (Systems Tool Kit) is also used to generate realistic dynamic topology and inter-satellite link (ISL) connectivity data for low-Earth orbit satellites. The agent continuously tries and fails in the simulation environment, storing the generated tuples in the replay buffer. During training, the Critic network randomly draws mini-batch data from this buffer to calculate the TD Error and update the gradients. The learning rate during training is 0.0005, the discount rate is 0.6, and the target network update coefficient is 0.005, so that the network updates are not too fast. The experience replay pool capacity is 10. 5 . Figure 3 This indicates that during the initial exploratory phase, at the beginning of training, the agent is completely unaware of the satellite network environment and engages in random exploration. The predicted rewards differ greatly from the actual rewards, resulting in very high loss values ​​accompanied by severe oscillations. During the rapid learning phase, as experience accumulates in the replay pool, the gradient descent algorithm takes effect, network parameters adjust rapidly, and the mean square value of the TD error decreases exponentially or stepwise. Finally, during the convergence phase, the loss value drops to near a lower baseline. Because the topology of low-Earth orbit satellite networks is dynamic, traffic flows are sudden, and the environment itself contains "unpredictable random noise," the loss value typically does not absolutely decrease to zero but rather fluctuates gently within a small range.

[0215] The low-Earth orbit satellite cooperative routing and resource scheduling method based on reinforcement learning and fuzzy control, as described in this embodiment, is briefly summarized as follows:

[0216] 1. Data Acquisition and Status Awareness

[0217] As the brain of the entire satellite network, the SDSN controller periodically collects real-time physical operational status data from the underlying low-Earth orbit satellite nodes. This data includes the satellite nodes' computing resources, buffer queue length, remaining battery power, as well as propagation delay, packet loss rate, and available bandwidth of adjacent inter-satellite links. This multi-dimensional raw physical data is then normalized to provide a standard input basis for subsequent intelligent algorithms.

[0218] 2. Fuzzy Control Evaluation and Reward Reshaping

[0219] Faced with complex, variable, and noisy physical data, the system utilizes a "fuzzy controller" to simulate human expert experience for evaluation. State dimensionality reduction involves inputting queuing delay, link load, and packet loss rate into the fuzzy system, which then performs inference through a pre-defined fuzzy rule base (IF-THEN) to output an intuitive "comprehensive fuzzy evaluation index" (reflecting the current link's condition). Based on this evaluation index, the "reward weights" of reinforcement learning are adjusted in real time. For example, when the network is extremely congested, the fuzzy controller automatically increases the weight of the "delay penalty," forcing the system to find alternative paths.

[0220] 3. DRL Joint Scheduling and Routing Decision

[0221] After obtaining the network topology and fuzzy evaluation, the reinforcement learning agent (Actor network) within the controller begins joint decision-making across layers. Feature extraction utilizes a Graph Convolutional Network (GCN) to process the complex satellite connectivity topology, while a Multilayer Perceptron (MLP) is used to handle traffic flow requirements and the fuzzy evaluation index. In a single forward computation process, the agent simultaneously outputs decisions from both layers:

[0222] Network layer (discrete): Which satellite is selected as the next-hop routing node for the data packet (based on Softmax probability).

[0223] Physical layer (continuous): Allocate bandwidth and transmit power to this service flow (based on the Sigmoid / Tanh function).

[0224] 4. Action delivery and closed-loop learning

[0225] After decision generation, the system enters the final stage of execution and self-optimization. Command issuance involves the SDSN controller converting routing tables and resource configuration parameters into underlying protocol commands, which are then sent to the corresponding satellite nodes for execution, and data packets are forwarded. Feedback and learning occur when the environmental state changes. The agent calculates the "instant reward" for this action based on the dynamic weights given by the fuzzy controller in "Step Two." The system stores these experiences in a playback pool and continuously updates the parameters of the Actor and Critic neural networks using the gradient descent algorithm by calculating the time difference (TD) error. As the model iterates and converges, the system gains the ability to adaptively output optimal routing and resource scheduling schemes within milliseconds when facing sudden traffic surges and drastic topology changes.

[0226] Other aspects of this embodiment can be found in the above method embodiments.

[0227] In summary, this invention discloses a method and system for cooperative routing and resource scheduling of low-Earth orbit (LEO) satellites based on reinforcement learning and fuzzy control. Through data acquisition and state awareness, the SDSN controller acts as the brain of the entire satellite network, periodically collecting real-time physical operating status data from the underlying LEO satellite nodes. This data includes the satellite nodes' computing resources, buffer queue length, remaining power, and propagation delay, packet loss rate, and available bandwidth of adjacent inter-satellite links. This multi-dimensional raw physical data is then normalized. Faced with complex, variable, and noisy physical data, the system utilizes a "fuzzy controller" to simulate human expert experience for evaluation. First, state dimensionality reduction is performed by inputting queuing delay, link load, and packet loss rate into the fuzzy system. Reasoning is then performed using a pre-set fuzzy rule base (IF-THEN), outputting an intuitive comprehensive fuzzy evaluation index. Finally, based on this evaluation index, the reward weights of reinforcement learning are adjusted in real-time. After obtaining the network topology and fuzzy evaluation, the reinforcement learning agent (Actor network) within the controller begins cross-layer joint decision-making. First, feature extraction is performed: a Graph Convolutional Network (GCN) is used to process the complex satellite connectivity topology, while a Multilayer Perceptron (MLP) is used to process traffic flow requirements and fuzzy evaluation indices. Then, actions are output synchronously. In a forward computation process, the agent simultaneously outputs decisions at both the network layer (discrete) and physical layer (continuous) levels. After decision generation, the system enters the final stage of execution and self-optimization. During instruction issuance, the SDSN controller converts the routing table and resource configuration parameters into underlying protocol instructions and issues them to the corresponding satellite nodes for execution, completing packet forwarding. The agent calculates the "instant reward" for this action based on the dynamic weights given by the fuzzy controller in "Step Two." The system stores these experiences in a replay pool and continuously updates the parameters of the Actor and Critic neural networks using gradient descent by calculating the time difference (TD) error. As the model iterates and converges, the system gains the ability to adaptively output optimal routing and resource scheduling schemes within milliseconds when facing sudden traffic surges and drastic topology changes. This invention overcomes the shortcomings of existing heuristic routing algorithms, which suffer from high signaling overhead and are prone to routing loops when facing highly dynamic star network topologies. Furthermore, while conventional reinforcement learning (RL) algorithms can approximate the optimal policy, they often suffer from slow convergence speed, difficulty in joint optimization of multiple objectives (such as throughput, latency, and power consumption), and policy oscillation in continuous action space when facing "fuzzy states" caused by sudden traffic or physical link fading.

[0228] The preferred embodiments and principles of the present invention have been described in detail above. For those skilled in the art, there may be changes in the specific implementation based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for cooperative routing and resource scheduling of low-Earth orbit satellites, characterized in that, Includes the following steps: S1, periodically collect real-time operational status information of the low-orbit satellite network, and normalize the real-time operational status information; S2, queuing delay, link load, and packet loss rate are selected from the normalized real-time operating status information as input variables for the fuzzy controller. Each input variable is mapped to low, medium, and high fuzzy sets. Mamdani fuzzy inference is performed using preset fuzzy rules, and the area centroid method is used to defuzzify and obtain the comprehensive fuzzy evaluation index I of the current inter-satellite link. eval ; Based on the comprehensive fuzzy evaluation index, delay weight, bandwidth weight, and energy consumption weight are generated through a preset dynamic transformation relationship. The delay penalty weight is increased when the link quality decreases, and the bandwidth-related weight is increased when the link quality improves. The energy consumption weight is adjusted according to the remaining energy status of the satellite node. Both the delay weight and the bandwidth weight increase as the comprehensive fuzzy evaluation index decreases, and the sum of the delay weight, bandwidth weight, and energy consumption weight is 1. S3. Construct a deep reinforcement learning joint scheduling and routing model. The state space of the model includes at least service flow information, a network topology matrix, and a comprehensive fuzzy evaluation vector composed of the comprehensive fuzzy evaluation index. The reward function is constructed based on the delay weight, bandwidth weight, and energy consumption weight. Graph structure features are extracted from the network topology matrix through the GCN branch, and features are extracted from the service flow information and the comprehensive fuzzy evaluation vector through the MLP branch. The outputs of the two branches are fused into a shared hidden layer state representation, and a joint action is output based on the shared hidden layer state representation. The joint action includes the next-hop routing node determined based on a discrete probability distribution, as well as continuous bandwidth allocation parameters and transmit power parameters. Before being executed, the joint action output by the deep reinforcement learning joint scheduling and routing model is first smoothed through a fuzzy confidence interval. S4, perform the joint action in the low-Earth orbit satellite network, collect environmental state transition feedback and immediate rewards, and update the network parameters of the deep reinforcement learning joint scheduling and routing model based on the feedback and immediate rewards.

2. The low-orbit satellite cooperative routing and resource scheduling method according to claim 1, characterized in that, In step S1, the real-time operating status information includes: the available computing resources, queue buffer length, and remaining energy of the satellite node, as well as the propagation delay, packet loss rate, and available bandwidth of adjacent inter-satellite links; the real-time operating status information is normalized and then used as the input of the fuzzy controller.

3. The low-orbit satellite cooperative routing and resource scheduling method according to claim 2, characterized in that, The processing procedure of the fuzzy controller in step S2 includes: The queue buffer length, available bandwidth, and packet loss rate in the real-time running status information are mapped to the membership degree in a fuzzy set. The trigger strength is calculated based on a preset fuzzy rule base, and a fuzzy set is aggregated and output. The comprehensive fuzzy evaluation index is obtained by unfuzzification using the area centroid method. The delay weight, bandwidth weight, and energy consumption weight for reinforcement learning are dynamically generated based on the comprehensive fuzzy evaluation index, and the sum of each weight is 1.

4. The low-Earth orbit satellite cooperative routing and resource scheduling method according to claim 3, characterized in that, In step S3, the deep reinforcement learning joint scheduling and routing model is constructed based on a Markov decision process, and the specific process is as follows: The state space includes: service flow demand information, the entire network connectivity topology matrix, and the comprehensive fuzzy evaluation index output in step S2; The combined action includes: discrete next-hop routing node selection identifiers, continuous allocated bandwidth values, and transmit power levels; The reward function is a weighted sum of latency cost, throughput gain and energy cost, and the corresponding latency weight, bandwidth weight and energy weight are output by step S2.

5. The low-Earth orbit satellite cooperative routing and resource scheduling method according to claim 4, characterized in that, In step S3, the forward propagation process of the deep reinforcement learning joint scheduling and routing model includes: Spatial features of the entire network connectivity topology matrix are extracted using a graph convolutional neural network, and vector features of business flow and comprehensive fuzzy evaluation index are extracted using a multilayer perceptron. The spatial features and the vector features are then fused into hidden layer features. The next-hop route is determined by selecting the probability distribution of each adjacent node through the output of the Softmax layer; at the same time, the bandwidth normalization factor is output through the Sigmoid layer and the power normalization factor is output through the Tanh layer. After physical dimension restoration, the allocated bandwidth value and the transmit power level are obtained.

6. The low-Earth orbit satellite cooperative routing and resource scheduling method according to claim 5, characterized in that, The network parameters are updated using an actor-critic architecture and a temporal difference algorithm. The specific process is as follows: Store the experience tuples generated by the interaction into the experience replay pool. The time difference error is calculated using mini-batch samples to update the value network; The policy gradient is calculated based on the advantage function, and the parameters of the deep reinforcement learning joint scheduling and routing model are updated. At the same time, the target network is introduced and a soft update mechanism is adopted to maintain training stability.

7. A low-Earth orbit satellite cooperative routing and resource scheduling system, used to perform the method as described in any one of claims 1-6, characterized in that, Includes the following modules: The status awareness module is used to periodically collect real-time operational status information of the low-Earth orbit satellite network and perform normalization processing. The fuzzy evaluation module is used to perform fuzzy inference and defuzzification with queuing delay, link load, and packet loss rate as inputs, to obtain the comprehensive fuzzy evaluation index I. eval And based on the comprehensive fuzzy evaluation index, delay weight, bandwidth weight and energy consumption weight are generated; The deep reinforcement learning decision module is used to take the service flow information, network topology matrix and the comprehensive fuzzy evaluation index as state information, construct a reward function with delay weight, bandwidth weight and energy consumption weight, process the network topology matrix through the GCN branch, process the service flow information and comprehensive fuzzy evaluation vector through the MLP branch and perform feature fusion, and output the next-hop routing node, bandwidth allocation parameters and transmit power parameters based on the fused shared hidden layer state representation. The execution and closed-loop training module is used to issue routing and resource configuration instructions corresponding to the joint actions to the underlying satellite nodes, collect environmental state transition feedback and real-time rewards, and update the network parameters of the deep reinforcement learning decision module.

Images