Satellite communication network controller dynamic deployment method

Abstract

The application relates to a satellite communication network controller dynamic deployment method, which comprises the following steps: according to a satellite network topology and node link traffic, the transmission time delay of a node to a node along a path in the network, path reliability and node load are calculated; a spectrum clustering method is used to divide the satellite network into multiple sub-networks; control right switching and controller migration are carried out in the sub-networks according to network topology and service traffic changes; a linear integer programming method is used to construct a control right switching model based on delay reliability constraints, and the mapping relationship between the satellite nodes and the controllers is dynamically adjusted; a controller migration model of a distributed partially observable Markov decision process is constructed, and the position of a control function node is dynamically migrated through a multi-agent reinforcement learning algorithm, so that the network benefit function reward value is maximized. The application realizes control link reliability improvement and control time delay reduction, and provides a data basis for real-time and flexible satellite network management and control.

Description

Technical Field

[0001] This invention relates to the field of satellite communication and network technology, specifically to a dynamic deployment method for satellite communication network controllers based on multi-agent reinforcement learning. Background Technology

[0002] Satellite networks are complex systems that use satellite platforms as carriers, combined with ground gateways and data centers, to achieve information collection, processing, and transmission. They feature wide-area coverage and ubiquitous access, playing an increasingly important role in fields such as ocean navigation and emergency communications. As a crucial component of future space communication infrastructure, they have become a cutting-edge research area. However, with the rapid development of the Internet, the amount of data in satellite networks is increasing exponentially. The centralized control method using the system control segment lacks flexibility in business management and cannot handle network faults in real time. Software-defined satellite networks (SDSN) have emerged as a new network architecture paradigm. They centralize control logic within the controller, with the data plane only responsible for traffic forwarding and scheduling, providing a technical foundation for flexible management and control of satellite networks. In SDN, controller deployment determines the number and location of controller nodes in the network. Controller deployment is a critical issue in SDN, directly affecting the reliability and effectiveness of the satellite network control plane.

[0003] In software-defined satellite networks (SDSNs), controller deployment is a process of dynamically configuring functional decisions within the control plane. The core objective is to improve control link reliability, reduce control latency between the controller and data plane nodes, and avoid controller overload by optimizing the location of control function nodes and the logically managed satellite node range. However, due to rapid changes in satellite network topology and limited onboard resources, dynamic controller deployment is necessary. This involves control handover and controller migration to continuously optimize the reliability, latency, and load performance of the SDSN control plane. Control handover involves transferring the logical ownership of satellite nodes within the control range from one controller to another without changing the location of the control function software. Controller migration involves changing the controller location, activating the control function software on another satellite node, and synchronizing network status data.

[0004] In SDSN, satellites move at high speeds, and network topology and traffic load are constantly changing. Existing controller deployment schemes and switch mapping relationships are insufficient to meet the performance requirements such as latency and reliability of the network control plane. It is necessary to dynamically migrate controllers and redistribute node control rights and controller-switch mapping relationships.

[0005] All nodes are potential controller migration targets. DQN, combining neural networks and Q-learning, obtains the optimal policy under the interaction of the minimum agent and the environment, but its output mechanism means that all outputs except the first one are no longer optimal. To address this issue, this invention models the dynamic migration problem of multiple controllers in satellite communication networks as a multi-agent reinforcement learning problem. It employs a dynamic deployment method for satellite communication network controllers based on Multi-Agent Reinforcement Learning (MARL), modeling satellite nodes as agents. Through cooperation among multiple satellite agents and interaction with the surrounding environment, the optimal controller migration policy is obtained. The MARL-based controller migration algorithm solves the dynamic deployment problem of controllers through collaborative decision-making by multiple agents based on the satellite network topology and service traffic. It adjusts the positions of control function nodes within sub-networks and dynamically migrates controllers to optimize control link reliability and latency under network topology changes. Furthermore, MARL is adaptable to multi-controller architectures, with each controller acting as an independent agent, making real-time decisions based on observations and avoiding centralized single-point bottlenecks. Simultaneously, it effectively avoids the NP-hard combinatorial explosion problem caused by excessively large state and action space dimensions by utilizing partial observation data within each sub-network, thus improving computational efficiency. In addition, MARL can flexibly adjust targets such as reliability and latency through a single reward function.

[0006] Existing research on dynamic controller deployment in terrestrial networks often employs clustering and heuristic methods to optimize controller location. However, because this research primarily focuses on static topologies, and the large transmission latency of satellite-to-ground links leads to untimely synchronization of dynamic changes in network topology and service traffic, ground-based controllers suffer from poor real-time decision-making, failing to meet the flexible management requirements of satellite networks. Research on dynamic controller deployment in satellite networks considers multiple performance indicators such as latency and reliability under dynamic topologies. Related studies model the dynamic controller deployment problem as a Markov decision process, with satellite agents making centralized decisions based on global network state information. However, due to the large number of satellite nodes, long inter-satellite distances, and significant synchronization latency of global network state information, centralized controller collaborative decision-making is difficult. Therefore, this invention models the dynamic controller deployment problem as a distributed partially observable Markov decision process. Satellite agents make distributed decisions based on local network state information, solving the problem with a lower state space and action space combination dimension, thereby improving control link reliability and reducing control latency under dynamic changes in satellite networks.

[0007] Network latency, path reliability, and node load are crucial factors affecting the dynamic deployment of controllers. Onboard processing and forwarding capabilities are relatively weak, and inter-satellite distances are long, resulting in significant propagation latency. Under high load, queuing latency can reach hundreds of milliseconds, making timely control of network node switching difficult when control latency is high. Furthermore, inter-satellite links are susceptible to interruptions due to space environment factors; selecting highly reliable paths ensures error-free transmission of control information. Simultaneously, due to limited onboard computing bandwidth and other resources, the overhead of control handover and controller migration during dynamic controller deployment should be minimized to reduce the impact on service transmission. Summary of the Invention

[0008] In view of the above-mentioned technical problems, this invention proposes a dynamic deployment method for satellite communication network controllers, which can monitor the drought status from multiple perspectives, more accurately reveal the evolution characteristics of watershed hydrological conditions and their causes of change, and provide a more reasonable scientific basis for watershed water resource management and sustainable development goals.

[0009] The technical solution to the technical problem of this invention is: a method for dynamically deploying a satellite communication network controller, comprising the following steps:

[0010] Step S1: Based on the satellite network topology and node link traffic, calculate the path-to-path traffic in the network. From node To the node Transmission delay Path reliability and node load ;

[0011] Step S2: Divide the satellite network into multiple sub-networks using spectral clustering.

[0012] Step S3: Within the subnetwork, switch control and migrate controllers based on changes in network topology and service traffic.

[0013] Step S4: Construct a control handover model based on time delay reliability constraints using linear integer programming method, and dynamically adjust the mapping relationship between satellite nodes and controllers;

[0014] Step S5: Construct a controller migration model for a distributed partially observable Markov decision process. Dynamically migrate the position of control function nodes using a multi-agent reinforcement learning algorithm to maximize the network benefit function reward value.

[0015] According to a technical solution of the present invention, in step S1, along the path From node To the node The transmission delay Expressed as:

[0016]

[0017] in, Indicates telemetry path The latency includes that of adjacent nodes. To the node Propagation delay between Indicates adjacent nodes To the node Transmission delay, Represents a node Queue processing delay;

[0018] node Node load Defined as: ;

[0019] in, Represents a node Send to node The number of requests;

[0020] Along the path in the network From node To the node Path reliability Represented as:

[0021]

[0022] in, Indicates node reliability. This indicates the reliability of the link.

[0023] According to one technical solution of the present invention, in step S3, the control switching specifically includes:

[0024] Based on satellite network traffic change information, the control plane dynamically adjusts the mapping relationship between the controller and the satellite nodes, transferring the logical ownership of the satellite nodes within the control range from one controller to another.

[0025] The overhead of the control handover includes three parts: inter-subnetwork controller synchronization overhead and the overhead of establishing new control links within the two subnetworks. The inter-subnetwork controller synchronization overhead for the control handover between the original controller A and the new controller B is a function of the data to be transmitted and the propagation path between them, expressed as:

[0026]

[0027] in, The amount of data to be synchronized between controller A and controller B. To synchronize data, network computing resources are consumed. The overhead of computing resources per unit of transmission distance. This is the distance bandwidth overhead factor;

[0028] The total overhead of switching control is:

[0029]

[0030] in, The cost of recalculating and issuing the routing topology within the control domain for controller A is expressed as:

[0031]

[0032] in, For sub-networks The cost of updating the topology is the number of nodes in the subnetwork. and number of links The function, For controller node A to nodes within the control domain Bandwidth overhead associated with updating control data;

[0033] The overhead of recalculating and distributing the routing topology within the control domain for controller B.

[0034] According to one technical solution of the present invention, the reward value of the control switch Represented as:

[0035]

[0036] in, For the reward function, , and These are the transmission delay difference, path reliability difference, and node load difference after the switch of control, respectively.

[0037] According to one technical solution of the present invention, in step S3, the controller migration specifically includes:

[0038] Based on satellite network traffic and change information, the control plane dynamically adjusts the position of control function nodes, activates the control function software on another satellite node, synchronizes network status data, and establishes a new control link with the satellite node.

[0039] The overhead of the controller migration consists of two parts: data synchronization overhead between controllers within the sub-network and the overhead of establishing a new control link within the sub-network. Specifically, the data synchronization overhead between controllers within the sub-network... Represented as:

[0040] ;

[0041] in, To transfer the controller from the node Migrate to node The amount of data that needs to be synchronized To synchronize data, network computing resources are consumed. The overhead of computing resources per unit of transmission distance. For distance bandwidth overhead factor, For the node Migrate to node The distance;

[0042] Total overhead of controller migration Represented as:

[0043] in, Recalculate and distribute the cost of routing topology within the control domain for controller A.

[0044] According to one technical solution of the present invention, the reward value of the controller migration Represented as:

[0045]

[0046] in, , and These are the differences in transmission delay after migration, path reliability, and node load, respectively.

[0047] According to a technical solution of the present invention, the controller transfer model for constructing a distributed partially observable Markov decision process is wherein the state space, action space, and reward function of the controller transfer model are set as follows:

[0048] The state space includes the latency from a node to its sub-network controller node, link reliability, sub-network location topology information, and the number of satellite agents taking the same action;

[0049] The action space is for the role selection of satellite intelligent agent nodes. A value of 1 indicates that the node becomes a controller node, and a value of 0 indicates that the node becomes a switch node.

[0050] The reward function is set as follows:

[0051]

[0052] in, This represents the migration penalty factor, used to suppress frequent controller migrations.

[0053] According to one technical solution of the present invention, the network benefit function is used to measure the impact of reliability, latency, and load on satellite network performance, and the network benefit function of node i is defined as:

[0054]

[0055] in, , and These represent normalized reliability, latency, and load, respectively. , and This represents its weighting coefficient.

[0056] According to one technical solution of the present invention, the objective of the control handover model is to maximize the handover reward value, and the constraints include control link reliability being greater than the average and control delay being less than the average, specifically expressed as:

[0057] .

[0058] According to one technical solution of the present invention, the objective of the controller migration model is to maximize the migration reward value, and the constraints include control link reliability being greater than the average and control latency being less than the average, specifically expressed as:

[0059] .

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] Based on the concept of this invention, a dynamic deployment method for satellite communication network controllers is proposed. The dynamic deployment problem of the controller is divided into two parts: control switching of switching nodes and migration of control function nodes, simplifying the complexity of the dynamic deployment problem. In the satellite network, satellite nodes can only obtain partial network environment data within a certain range. The dynamic deployment problem of the controller is modeled as a partially visible Markov decision process. Based on the network environment data, multiple intelligent agent nodes collaboratively decide the position of the controller node. Finally, to address the problem that the reliability and effectiveness of the control plane in software-defined satellite networks decrease with changes in network topology, the MARL-DCP algorithm is used to dynamically allocate node control rights and adjust the position of control function nodes to maximize the dynamic deployment reward value of the controller, thereby realizing the dynamic deployment of the satellite network controller.

[0062] This invention utilizes dynamic migration of controller locations to select more reliable paths for transmitting control information. Simulation results demonstrate that, compared to static deployment (SCP), the method of this invention significantly improves control link reliability. In polar regions, dynamic deployment further optimizes reliability due to the close proximity of nodes and high link reliability; while in low-latitude regions, frequent switching and migration avoid reliability degradation caused by link interruptions.

[0063] This invention optimizes the control path through dynamic controller deployment, reducing propagation and queuing delays. Through multi-agent reinforcement learning, the controller position adapts to topology changes in real time, avoiding additional delays introduced by long-distance inter-satellite links.

[0064] In this invention, the controller migration algorithm prioritizes latency and reliability while distributing request traffic through a load balancing mechanism. In the event of a sudden surge in business traffic, dynamic deployment can quickly adjust the controller load and avoid network congestion caused by single-point overload.

[0065] This invention integrates the weighted sum of reliability, latency, and load in NUF, and dynamic deployment greatly improves the NUF return value compared to static deployment. By flexibly adjusting the weight coefficients, this invention can adapt to different satellite service needs (such as real-time communication and big data transmission) and achieve multi-objective optimization.

[0066] This invention employs a distributed decision-making mechanism to avoid centralized single points of failure, thereby improving network resilience. Multi-agent reinforcement learning enables the system to adapt to environmental changes based on local observation information, reducing the state space dimension and improving computational efficiency. Attached Figure Description

[0067] Figure 1 This is a schematic diagram illustrating the implementation process of a dynamic deployment method for satellite communication network controllers based on multi-agent reinforcement learning, provided as an example of the present invention.

[0068] Figure 2 A satellite network controller deployment scenario diagram provided for an example of the present invention;

[0069] Figure 3 A satellite network controller migration framework based on multi-agent reinforcement learning is provided for this example.

[0070] Figure 4 Satellite network control switching provided for an example of the present invention;

[0071] Figure 5 The satellite network control handover process provided for this invention example;

[0072] Figure 6 Satellite network controller migration for simulation verification of this invention;

[0073] Figure 7 The satellite network controller migration process is simulated and verified for this invention;

[0074] Figures 8(a) and 8(b) show the reliability comparison between dynamic deployment and static deployment verified by the simulation of the present invention;

[0075] Figures 9(a) and 9(b) show the comparison of dynamic and static deployment latency in the simulation verification of this invention;

[0076] Figures 10(a) and 10(b) show the comparison of dynamic and static deployment loads in the simulation verification of this invention;

[0077] Figures 11(a) and 11(b) show a comparison of the return values ​​of the network benefit function for dynamic and static deployments as verified by the simulation of this invention. Detailed Implementation

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0079] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0080] like Figures 1 to 1 As shown in 1(b), the dynamic deployment method for satellite communication network controllers based on multi-agent reinforcement learning of the present invention involves a satellite communication network. Includes space segment and ground segment network The two are connected via a satellite-to-ground link, such as Figure 2 As shown. The ground segment includes satellite gateways, switching nodes, and user nodes. Switching nodes are connected to each other via ground links, and users are connected to switching nodes via user ground links. The satellite gateway deploys the main controller. Develop satellite network forwarding strategies and upload them to the space segment's slave controller. Simultaneously, user nodes transmit data to the satellite network via a satellite gateway. The ground main controller is located in a fixed position, deployed at the ground network control center. (Satellite network) Composed of a mix of LEO satellites in polar and inclined orbits, with adjacent satellites connected by inter-satellite links, the system is divided into... Individual Networks and in sub-networks Deployment from controller Each switching node is connected to only one controller.

[0081] The present invention provides a dynamic deployment method for satellite communication network controllers based on multi-agent reinforcement learning, comprising the following steps:

[0082] Step S1: Based on the satellite network topology and node link traffic, calculate the path-to-path traffic in the network. From node To the node Transmission delay Path reliability and node load ;

[0083] Step S2: Divide the satellite network into multiple sub-networks using spectral clustering.

[0084] Step S3: Within the subnetwork, switch control and migrate controllers based on changes in network topology and service traffic.

[0085] Step S4: Construct a control handover model based on time delay reliability constraints using linear integer programming method, and dynamically adjust the mapping relationship between satellite nodes and controllers;

[0086] Step S5: Construct a controller migration model for a distributed partially observable Markov decision process. Dynamically migrate the position of control function nodes using a multi-agent reinforcement learning algorithm to maximize the network benefit function reward value.

[0087] Regarding satellite network latency, satellite communication networks need to transmit control information with low latency to meet the real-time and efficient control requirements of satellite networks. This invention considers the impact of propagation delay, transmission delay, and queuing processing delay on the effectiveness of satellite network control. (Adjacent nodes) To the node The time delay between them includes the propagation delay. ,in Adjacent nodes To the node The distance between them Let it be the speed of light. Adjacent nodes To the node The amount of data sent between them For the data transmission rate, then the adjacent nodes To the node The transmission delay is .node The queuing processing delay is , Represents a node queue length, For arrival rate. Define the rate along the path in the network. From node To the node latency Expressed as follows

[0088] (1)

[0089] in, The number of inter-satellite links along the path is . For path The number of nodes on it is .

[0090] Regarding satellite path reliability, higher path reliability means a longer time for error-free transmission of control information between network nodes. Therefore, control links should select highly reliable paths to ensure the delivery of control information. Path reliability consists of node reliability and link reliability, along the path... By node To the node Path reliability Defined as

[0091] (2)

[0092] Among the nodes The failure probability is ,link Failure probability .

[0093] Because controller resources are limited, only a portion of requests from other satellite nodes can be processed within a given timeframe. Furthermore, the probability of controller failure increases with the number of requests. Excessive controller load on the control plane can negatively impact the performance of the entire network. Therefore, when deploying controllers, the load between controllers must be considered to ensure overall network performance. The load is defined as:

[0094] (3)

[0095] in, For nodes Send to node The number of requests, and the maximum load is:

[0096]

[0097] node Normalized load balancing is .

[0098] In SDN-based satellite networks, multi-controller deployment presents a joint optimization problem addressing reliability, latency, and load. This invention proposes a Network Benefit Function (NUF) to measure the impact of multiple factors on the performance of LEO satellite networks. Node NUF is

[0099] (4)

[0100] in, , and These represent normalized reliability, latency, and load, respectively. , and This represents its weighting coefficient. The location of the controller deployment determines the control latency, the control domain partitioning ensures reliability, and further affects node load; these three factors together define the control plane performance. Furthermore, reliability, latency, and load metrics exhibit convex characteristics; a weighted sum of these three can preserve these convex characteristics and guarantee gradient descent. Simultaneously, the metrics can be adjusted through weighting coefficients to adapt to differences in satellite services, explicitly linking controller deployment to the network benefit function. The normalized path latency is...

[0101] (5)

[0102] in Let be the maximum path delay of the network. Assume the node... reachable subnetwork The number of other nodes is Then the node The overall reliability is the average of its reliability to all nodes within the subnetwork, which is:

[0103] (6)

[0104] To better measure satellite nodes The reliability is represented by normalized reliability.

[0105] (7)

[0106] in, It represents the maximum average reliability of the satellite topology at a given time slice.

[0107] The overhead of control handover includes three parts: inter-subnetwork controller synchronization overhead and the overhead of establishing new control links within the two subnetworks. The inter-subnetwork controller synchronization overhead for control handover between the original controller A and the new controller B is expressed as a function of the data to be transmitted and the propagation path between them, and is represented as...

[0108] (8)

[0109] in The amount of data to be synchronized between controller A and controller B. To synchronize data, network computing resources are consumed. The overhead of computing resources per unit of transmission distance. This is the distance bandwidth overhead factor.

[0110] Control switching involves transferring the logical ownership of satellite nodes within the control range from one controller to another without changing the location of the control function software. This is also known as "switch migration." Therefore, the total overhead of switch migration (control switching) is...

[0111] (9)

[0112] in The cost of recalculating and distributing the routing topology within the control domain for controller A is defined as follows:

[0113] (10)

[0114] in For sub-networks The cost of updating the topology is the number of nodes in the subnetwork. and number of links The function; the latter half is the function from controller node A to nodes within the control domain. The bandwidth overhead of updating control data.

[0115] Similarly, The cost of recalculating and issuing the routing topology within the control domain for controller B is the same.

[0116] Define nodes Switching report from controller A to controller B for

[0117] (11)

[0118] in For the reward function, , and These are the latency, reliability, and load difference after control switchover.

[0119] During controller migration, the switch-controller mapping table and control policy information of subnetwork A are migrated from the original controller i to the candidate controller j. The transmission path is Its expenses are

[0120] (12)

[0121] in, To transfer the controller from the node Migrate to node The amount of data that needs to be synchronized To synchronize data, network computing resources are consumed. The overhead of computing resources per unit of transmission distance. For distance bandwidth overhead factor, For the node Migrate to node The distance. Simultaneously, a new control link is established and telemetry is performed within the subnetwork. The controller migration overhead is expressed as...

[0122] (13)

[0123] in, Recalculate and distribute the cost of routing topology within the control domain for controller A.

[0124] Nodes are defined by the controller Migration to controller Migration reward value for

[0125] (14)

[0126] in , and These are the post-migration latency, reliability, and load difference, respectively.

[0127] In step S5, a controller migration model of a distributed partially observable Markov decision process is constructed. The position of the control function node is dynamically migrated through a multi-agent reinforcement learning algorithm to maximize the reward value of the network benefit function.

[0128] Due to the large scale and wide coverage of satellite communication networks, controller nodes often struggle to obtain real-time position and traffic data of all global satellite nodes within a given timeframe. Therefore, controller migration within a subnetwork is merely a distributed decision based on local observation information. Consequently, multi-agent collaborative decision-making only obtains state data from a subset of satellite nodes. This application models the controller migration problem as a Distributed Partially Observable Markov Decision Process (DPOMDP). DPOMDP is an extension of the Markov Decision Process model, where system information is partially known to individual agents, and agents make decisions based on incomplete current state information, expressed as:

[0129] (15)

[0130] in The total number is Multi-agent groups; Let be the state space, representing a series of states of the agent; Indicates the first All possible actions of an intelligent agent; Let be the state transition probability. Indicates the state as The agent selects actions When, the state transitions to The probability of; Indicates the first The reward function of an agent The state is intelligent agents Select Action Instantaneous return; Compared to Markov decision processes, DPOMDP has a significantly reduced observation space dimension, thus improving computational efficiency. Defined in the execution action And reach the state Post-Intelligent Agent Received observation The probability of; This serves as a discount factor to balance immediate rewards with long-term benefits. This section, tailored to the characteristics of software-defined satellite networks, sets the status, actions, and rewards as follows.

[0131] (1) State Space: Reflects the satellite network state and controller performance changes. Satellite agents select appropriate actions based on the state. Requests from switch nodes must reach and be processed by their respective controller nodes. When the latency between the two is too high, the controller load is too high, or the transmission link reliability is low, the switch node cannot be managed by the original controller node. Satellite agent nodes The state space is defined as

[0132] (16)

[0133] in For nodes To the sub-network Controller Node The time delay, For nodes To the sub-network Controller Node Link reliability, For sub-network location topology information, For intelligent agents The number of satellite agents that take the same action.

[0134] (2) Action Space: This section reduces the satellite network controller migration problem to a role switching problem. The role selection of satellite agent nodes is a discrete variable, with a value of 1 indicating becoming a controller node and a value of 0 indicating becoming a switch node. Therefore, the satellite agent node... Action space is defined as

[0135] (17)

[0136] In subnetwork In this scenario, only one agent node has an action value of 1, while the actions of all other agents have an action value of 0. Furthermore, for the same agent node, if its result in two adjacent time slices is that it selects to become the controller node, it means that no controller migration has occurred in the subnetwork, only a switch of control has taken place.

[0137] (3) Reward Function: The goal of controller migration in this section is to maximize the migration reward value within the subnetwork. Therefore, the reward function for controller migration in the satellite network is set as follows:

[0138] (18)

[0139] in This is a migration penalty factor used to suppress frequent controller migrations.

[0140] Each satellite node is considered an intelligent agent, and its state includes information such as network topology and node traffic load. The mapping relationship between the controller and the switch is dynamically adjusted based on the current network state. For example... Figure 3 As shown, the state space, action space, and reward function are designed according to network performance requirements to guide the satellite agent in making optimization decisions.

[0141] Satellite network control switch

[0142] To address the changes in the number of switch nodes in subnetworks caused by changes in satellite network topology, satellite switches need to be... The switch is separated from the original controller A and assigned to a new controller B. This process is divided into two parts: switch migration out and migration in. Figure 4 As shown. When the satellite node in the controller A sub-network Move to the edge, into the range of controller B sub-network. If the reliability of the control link is greater than that of the original control link, then the satellite node... The new satellite node is incorporated into the control domain of controller B, and a new control link is established with controller B. Simultaneously, within control domain B, the location and traffic information of the newly accessed satellite node are acquired, the network topology and routing are updated, and the switch migration is completed. Existing switch nodes in the subnetwork... If the satellite node exceeds the control range of controller A, the original control link will be disconnected, and the satellite node will be... Remove the subnet, reroute subnet A, and complete the switch migration.

[0143] In addition, if the switch When simultaneously within the control range of multiple controllers, to address the issue of increased control latency between switches and controllers in subnetworks caused by changes in service traffic within the satellite network, the switches... Control is switched from the original controller to the controller with the highest Network Benefit Function (NUF). Calculate the switch values ​​separately. The latency and link reliability to other controllers are assessed to determine the reward value for switching control to the corresponding controller. After weighing the migration overhead against the reward value, the switch is moved to the controller node with the highest reward value. (Switch) The goal of the switch of control is

[0144] (19)

[0145] This means maximizing the switch migration reward value, and ensuring that the control link reliability of the migrated controllers A and B is greater than the average. Control delay is less than the average value .

[0146] The dynamic control handover algorithm consists of two parts: determining the switch node to be switched and the control handover process. First, based on the traffic load status in the satellite network, the switch node requiring switching due to performance indicators such as latency, traffic load, or reliability is determined using the control handover model. Secondly, computing switches Control was transferred from the original controller Switch to adjacent controller The cost is considered, and the switching option that maximizes the switching reward is selected. If the switching reward is positive, a switch of control is initiated, as follows: Figure 5 As shown; if the migration report value is negative, it means that switching to an adjacent controller node cannot meet the latency reliability requirements, and controller migration is required.

[0147] Switching reward value exceeds threshold Under the condition that the constraints are met, control of the node is switched from controller A to controller B. Controller A sends a handover request command to controller B, including the node's status and link information. Upon receiving the command, controller B sends a network entry request to the node, and after replying, sends a start migration command to A. Controller A removes control over the node, and the node sends a control transfer request to controller B. The node establishes a control link with controller B and disconnects the controller link with controller A. Controllers A and B then perform route calculations and updates within the subnetwork, respectively. After distributing the routing table to the node, controller B sends a handover end command to controller A and simultaneously reports to the ground master controller, completing the control handover.

[0148] Satellite Network Controller Migration

[0149] When the controller, due to dynamic changes in network topology or sudden traffic surges, no longer meets the performance requirements of the subnetwork control information transmission link in terms of reliability and latency—that is, when the mapping relationship between the controller of subnetwork A and other satellite switching nodes cannot achieve real-time and reliable request forwarding—the controller needs to migrate within the subnetwork, activating the control function on another satellite node and synchronizing network status data, such as... Figure 6 As shown. Based on the satellite network topology and traffic status, the target node is determined, network control functions are migrated to this node, control links with other switches are re-established, effective control of the node is achieved, the controller migration is completed, and the reliability and effectiveness of network control are ensured. The target of the controller migration in this invention is...

[0150] (20)

[0151] That is, to maximize the controller migration reward value, and for controller nodes Migrate to controller node Subsequently, the reliability of the control link within control domain A is greater than the average value. Control delay is less than the average value .

[0152] The controller migration algorithm consists of two parts: determining the target controller node and performing the controller migration. Based on satellite network topology, node link traffic, and other data, the target controller node that needs to be migrated due to changes in traffic and topology is determined. The calculation is performed by the original controller node. Migrate to target controller node The overhead and migration reward value are calculated. If the migration reward value is negative, control domain repartitioning is performed; otherwise, controller migration is performed, the control function software is activated on another satellite node, and network status data is synchronized. The process is as follows: Figure 7 As shown.

[0153] If the migration reward is positive and the constraints are met, control of subnetwork A is transferred to the original controller node. Migrate to target controller node .node To the node Send a migration request, including the node's status and link information. Node Upon receiving, it will be sent to the node. Send start migration command, node Send subnetwork status data and a mapping table of controllers and switches. Node Upon receiving the route, the node calculates and updates the subnet route, and then distributes the routing table. After confirming receipt of the routing table, the node... Establish a new control link within the subnetwork, node Clear the existing control link. Last node. Send a migration end command and report to the ground main controller to complete the controller migration.

[0154] The present invention discloses a dynamic deployment method for satellite communication network controllers based on multi-agent reinforcement learning. The dynamic deployment problem is divided into two parts: control switching of switching nodes and migration of control function nodes, simplifying the complexity of the dynamic deployment problem. Considering that satellite nodes in a satellite network can only obtain partial network environment data within a certain range, the dynamic deployment problem is modeled as a partially visible Markov decision process. Based on the network environment data, multiple agent nodes collaboratively decide the location of the controller nodes. Finally, addressing the issue that the reliability and effectiveness of the control plane in software-defined satellite networks decrease with changes in network topology, the MARL-DCP algorithm is used to dynamically allocate node control rights and adjust the location of control function nodes to maximize the reward value of dynamic controller deployment.

[0155] Simulation Analysis

[0156] In this invention, the simulation analysis uses a satellite network consisting of 293 LEO satellites in inclined and polar orbits. The inclined orbits have an altitude of 550 km and an inclination of 50°, with 11 satellites evenly distributed across 19 orbital planes; the polar orbits have an altitude of 560 km and an inclination of 86.5°, with 14 satellites evenly distributed across 6 orbital planes. To calculate the reliability of the satellite network, the failure rates of links and nodes are randomly set to [0.05, 0.2]. To verify the feasibility and effectiveness of the proposed dynamic deployment method (DCP) for satellite communication network controllers based on multi-agent reinforcement learning, simulations comparing it with a static controller deployment method are performed over 11 consecutive time slices, with an interval of 100 s between each time point.

[0157] Figures 8(a) and 8(b) show the reliability comparison between dynamic and static deployments verified by the simulation of this invention; Figures 9(a) and 9(b) show the latency comparison between dynamic and static deployments verified by the simulation of this invention; Figures 10(a) and 10(b) show the load comparison between dynamic and static deployments verified by the simulation of this invention; Figures 11(a) and 11(b) show the network benefit function return values ​​comparison between dynamic and static deployments verified by the simulation of this invention. In Figures 8(a) to 11(b), the bars sloping from right to left in the bar chart correspond to the DCP algorithm proposed in this application, and are usually located on the left side of two opposite bars. The bars sloping from left to right in the bar chart correspond to the SCP algorithm, and are usually located on the right side of two opposite bars.

[0158] Figure 8 shows the results regarding the reliability of the telemetry control link. Compared with the static deployment-based dynamic allocation controller deployment (SCP) algorithm, the DCP proposed in this application has a lower control link reliability because the SCP algorithm selects the optimal controller deployment location under multi-time-slice aggregation. Figure 8(a) shows the average reliability of the 17 sub-network controllers at 11 time points. Due to the DCP changing the controller's location, its controller reliability is higher during the simulation period, with an average improvement of about 50% compared to SCP. However, when the sub-network operates in polar regions, the link reliability is relatively higher than in low-latitude regions due to the closer distance between nodes, and the number of SCP control handovers is reduced, resulting in higher control plane reliability compared to DCP. Figure 8(b) shows the average reliability return value of the 17 sub-networks under time slices, representing the reliability difference between adjacent time slices using different controller deployment methods. Due to the migration of the control function nodes in DCP, the number of switch handovers is less than that of SCP, and the increase in the average reliability return value of the dynamic deployment algorithm is greater than that of the static deployment algorithm. Furthermore, the number of switch handovers between different time slices and sub-networks in DCP fluctuates less, and the controller reliability and its reward value are more balanced compared to the frequent handovers in SCP. The proposed algorithm can effectively improve the reliability of the controller, reduce the reliability loss of the control link during dynamic deployment, and ensure the resilience of the control plane.

[0159] Regarding control latency, Figure 9(a) shows the average latency of each sub-network controller over consecutive time slices. Compared to SCP, the DCP-based method for dynamically migrating controller locations selects nodes with lower latency within the sub-network as migration targets, effectively reducing the average control link latency by 20%. Furthermore, the control latency fluctuations between different sub-networks are smaller, effectively balancing control latency. Figure 9(b) shows the average latency reward value over consecutive time slices. Since DCP represents the optimal controller deployment location for each sub-network, its latency reward value in adjacent time slices is smaller. However, the latency reward value of DCP is still greater than that of SCP throughout the entire simulation period, and the latency reward value remains stable over time. In other words, the proposed algorithm can effectively reduce controller latency, reduce control link latency during dynamic deployment, and minimize control plane interruption time.

[0160] Regarding controller load, Figure 10(a) shows the average load of each sub-network controller over consecutive time slices. In the sub-network, DCP prioritizes ensuring latency and reliability rewards, resulting in a relatively high controller load. When DCP dynamically migrates node control functions, the controller migration load increases due to short-term control plane interruptions and routing table updates. Regarding load rewards, Figure 10(b) shows the average load reward of sub-network controllers over consecutive time slices. DCP improves upon SCP by more than 10% on average, meaning the proposed algorithm can reduce controller load to some extent and provide stable load rewards during continuous controller migrations, mitigating network congestion and other faults caused by excessive load.

[0161] Regarding the network benefit function return value, as shown in Figure 11, the dynamic controller deployment algorithm is the optimal deployment method for the network benefit function within each time slice. Figure 11(a) shows the average NUF of each sub-network controller under continuous time slices. Compared with SCP, which deploys controllers based on node reliability, latency, and load throughout the simulation time, the NUF value of the DCP controller is significantly better than that of static controller deployment. Furthermore, due to the flexible switching of control rights and adaptive migration of control function nodes, as shown in Figure 11(b), DCP improves the average NUF return value of sub-networks under continuous time slices by approximately 10% compared to SCP, achieving controller deployment that maximizes the network benefit function return value within a single time slice. However, DCP requires migrating the locations of some sub-network control function nodes and synchronizing network status data, achieving a greater NUF improvement with a larger bandwidth consumption compared to SCP. The proposed algorithm improves the satellite network NUF within continuous time slices and comprehensively optimizes the reliability, latency, and controller load of the control link during the dynamic deployment of the satellite network controller, effectively ensuring the reliability and effectiveness of satellite network control under dynamic network topology and bursty service traffic.

[0162] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a dynamic deployment method for a satellite communication network controller as described in any of the above technical solutions.

[0163] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0164] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the terminal device. It can also be used to temporarily store data that has been output or will be output.

[0165] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a dynamic deployment method for a satellite communication network controller as described in any of the above technical solutions.

[0166] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. They can be implemented using computer-executable program code, thus allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this invention is not limited to any particular hardware and software combination.

[0167] In summary, the present invention provides a dynamic deployment method for satellite communication network controllers based on multi-agent reinforcement learning. This method divides the dynamic deployment problem into two parts: control switching of switching nodes and migration of control function nodes, simplifying the complexity of the dynamic deployment problem. Since satellite nodes in a satellite network can only acquire partial network environment data within a certain range, the dynamic deployment problem is modeled as a partially visible Markov decision process. Based on the network environment data, multiple agent nodes collaboratively decide the location of the controller nodes. Finally, addressing the issue that the reliability and effectiveness of the control plane in software-defined satellite networks decrease with changes in network topology, the MARL-DCP algorithm is used to dynamically allocate node control rights and adjust the location of control function nodes, maximizing the dynamic deployment reward value of the controller and achieving dynamic deployment of the satellite network controller.

[0168] Furthermore, it should be noted that the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0169] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0172] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for dynamically deploying a satellite communication network controller, characterized in that, Includes the following steps: Step S1: Based on the satellite network topology and node link traffic, calculate the path-to-path traffic in the network. From node To the node Transmission delay Path reliability and node load ; Step S2: Divide the satellite network into multiple sub-networks using spectral clustering. Step S3: Within the subnetwork, switch control and migrate controllers based on changes in network topology and service traffic. Step S4: Construct a control handover model based on time delay reliability constraints using linear integer programming method, and dynamically adjust the mapping relationship between satellite nodes and controllers; Step S5: Construct a controller migration model for a distributed partially observable Markov decision process. Dynamically migrate the position of control function nodes using a multi-agent reinforcement learning algorithm to maximize the network benefit function reward value.

2. The method according to claim 1, characterized in that, In step S1, along the path From node To the node The transmission delay Expressed as: in, Indicates telemetry path The latency includes that of adjacent nodes. To the node Propagation delay between Indicates adjacent nodes To the node Transmission delay, Represents a node Queue processing delay; node Node load Defined as: ; in, Represents a node Send to node The number of requests; Along the path in the network From node To the node Path reliability Represented as: in, Indicates node reliability. This indicates the reliability of the link.

3. The method according to claim 2, characterized in that, In step S3, the control switch specifically includes: Based on satellite network traffic change information, the control plane dynamically adjusts the mapping relationship between the controller and the satellite nodes, transferring the logical ownership of the satellite nodes within the control range from one controller to another. The overhead of the control handover includes three parts: inter-subnetwork controller synchronization overhead and the overhead of establishing new control links within the two subnetworks. The inter-subnetwork controller synchronization overhead for the control handover between the original controller A and the new controller B is expressed as a function of the data to be transmitted and the propagation path between them: in, The amount of data to be synchronized between controller A and controller B. To synchronize data, network computing resources are consumed. The overhead of computing resources per unit of transmission distance. This is the distance bandwidth overhead factor; The total overhead of switching control is: in, The cost of recalculating and issuing the routing topology within the control domain for controller A is expressed as: in, For sub-networks The cost of updating the topology is the number of nodes in the subnetwork. and number of links The function, For controller node A to nodes within the control domain Bandwidth overhead associated with updating control data; The overhead of recalculating and distributing the routing topology within the control domain for controller B.

4. The method according to claim 3, characterized in that, The reward value of the control transfer Represented as: in, For the reward function, , and These are the transmission delay difference, path reliability difference, and node load difference after the switch of control, respectively.

5. The method according to claim 3, characterized in that, In step S3, the controller migration specifically includes: Based on satellite network traffic and change information, the control plane dynamically adjusts the position of control function nodes, activates the control function software on another satellite node, synchronizes network status data, and establishes a new control link with the satellite node. The overhead of the controller migration consists of two parts: data synchronization overhead between controllers within the sub-network and the overhead of establishing a new control link within the sub-network. Specifically, the data synchronization overhead between controllers within the sub-network... Expressed as: ； in, To transfer the controller from the node Migrate to node The amount of data that needs to be synchronized To synchronize data, network computing resources are consumed. The overhead of computing resources per unit of transmission distance. For distance bandwidth overhead factor, For the node Migrate to node The distance; Total overhead of controller migration Represented as: in, Recalculate and distribute the cost of routing topology within the control domain for controller A.

6. The method according to claim 5, characterized in that, The reward value of the controller migration Represented as: in, , and These are the differences in transmission delay after migration, path reliability, and node load, respectively.

7. The method according to claim 2, characterized in that, The controller transfer model for constructing a distributed partially observable Markov decision process is defined as follows: the state space, action space, and reward function of the controller transfer model are set as follows: The state space includes the latency from a node to its sub-network controller node, link reliability, sub-network location topology information, and the number of satellite agents taking the same action; The action space is for the role selection of satellite intelligent agent nodes. A value of 1 indicates that the node becomes a controller node, and a value of 0 indicates that the node becomes a switch node. The reward function is set as follows: in, This represents the migration penalty factor, used to suppress frequent controller migrations.

8. The method according to claim 2, characterized in that, The network benefit function is used to measure the impact of reliability, latency, and load on satellite network performance. The network benefit function for node i is defined as follows: in, , and These represent normalized reliability, latency, and load, respectively. , and This represents its weighting coefficient.

9. The method according to claim 2, characterized in that, The objective of the control handover model is to maximize the handover reward value. Constraints include control link reliability being greater than average and control latency being less than average. The specific expression is as follows: 。 10. The method according to claim 2, characterized in that, The objective of the controller migration model is to maximize the migration reward value. Constraints include control link reliability being greater than average and control latency being less than average. The specific expression is as follows: 。

Images