A multi-agent hybrid routing method and system based on direction perception assistance

Abstract

The application provides a multi-agent hybrid routing method and system based on direction perception assistance, which comprises the following steps: in a pre-built satellite network environment model, state information in a domain is collected by a domain manager, and direction perception information and inter-domain state information are collected from adjacent inter-domain nodes by a flooding mechanism of an inter-domain manager; a core control satellite receives a data packet to be forwarded and a corresponding routing request; the data packet comes from each satellite in a sub-domain or a core control satellite of another sub-domain; an inter-domain routing stage utilizes a deep reinforcement learning neural network to make a cross-domain routing decision, and a centralized routing method is used to make an intra-domain routing decision when the data packet reaches a target domain; and in a reward function of the deep reinforcement learning neural network based on direction perception assistance, direction perception information and state information are integrated to optimize the routing decision. The application can improve adaptability and global optimization capability in a complex satellite network communication environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of satellite networks and industrial internet, and particularly relates to a multi-agent hybrid routing method and system based on direction perception assistance. BACKGROUND

[0002] With the continuous surge in global communication demand, the deep integration of low earth orbit (LEO) satellite networks with existing ground networks provides an unprecedented opportunity for the expansion of communication coverage for the sixth generation of mobile communication (6G) networks. However, the highly dynamic nature of LEO satellite nodes poses a serious challenge to addressing and routing strategies for large constellations. Among them, in the low earth orbit environment, the addressing and routing strategy for building a large constellation network is to: ① determine the address of each satellite; ② determine the transmission path of the data packet in the network.

[0003] In the prior art, in order to solve the above challenges, the routing technology solutions include:

[0004] (1) Centralized routing method, relying on a software-defined network (SDN) controller as the core hub, effectively separates the control plane from the data plane. However, as the network size grows, the management complexity and resource overhead related to flow_mod message passing significantly increase, becoming a major bottleneck to its application.

[0005] (2) Distributed routing method, each satellite node independently collects surrounding network information and makes routing decisions accordingly. Although this method gives the network a high degree of flexibility, excessive reliance on local information can cause the routing decision to deviate from the global optimal solution, creating an irreconcilable contradiction between the acquisition and management granularity of network state information.

[0006] Therefore, how to provide a routing method that can overcome the centralized routing method and the distributed routing method, and improve the adaptability and global optimization capability in a complex satellite network communication environment, is a technical problem that needs to be solved. SUMMARY

[0007] In view of this, the embodiments of the present application provide a multi-agent hybrid routing method and system based on direction perception assistance to eliminate or improve one or more defects in the prior art.

[0008] In one aspect of the present application, a multi-agent hybrid routing method based on direction perception assistance is provided. A satellite network is pre-divided into multiple sub-domains, each containing a core control satellite. The core control satellite is used as a node of the satellite network, and a domain manager and an inter-domain manager are pre-deployed on the core control satellite. The method comprises: in a pre-built satellite network environment model, collecting state information within the domain using the domain manager, and collecting direction perception information and inter-domain state information from adjacent inter-domain nodes using the inter-domain manager through a flooding mechanism; wherein the state information includes multiple types of link bandwidth, link delay, link load, and network topology information; the core control satellite receives data packets and corresponding routing requests that need to be forwarded; the data packets come from each satellite in the sub-domain or the core control satellite of other sub-domains; when the core control satellite analyzes the routing request and determines that the data packet is in the inter-domain routing stage, a pre-trained deep reinforcement learning neural network based on direction perception assistance contained in the inter-domain manager is used to make cross-domain routing decisions; when the core control satellite analyzes the routing request and determines that the data packet reaches the target domain, the domain manager deployed on the core control satellite is used to make intra-domain routing decisions according to a centralized routing method; wherein the direction perception information and the state information are integrated in the reward function of the deep reinforcement learning neural network based on direction perception assistance to optimize the routing decision.

[0009] In some embodiments of the present application, the method further comprises: before making the routing decision, pre-building a satellite network environment model for a low earth orbit satellite network, and generating a state space after the satellite network environment model is built, the state space containing the state information within the domain, the inter-domain state information, and the direction perception information; during the routing process, for a newly arrived routing request, recording service request information, the service request information containing multiple types of source node, destination node, duration, and service requirement width.

[0010] In some embodiments of the present application, the direction perception information is derived based on a location-assisted routing protocol through historical trajectory parameters and historical link quality data of the satellite; wherein the historical link quality data includes link delay and link bandwidth.

[0011] In some embodiments of the present application, the direction perception information fuses the included angle and direction information, which is defined in the formula as follows:

[0012] D i，k =cos(θ i ,k)·sgn((P k -P i )×(P D -P i ));

[0013] wherein D i，kIt is direction-aware information, θ i k is the direction vector from the current control node i to the neighboring domain node k. and the direction vector from the current control node i to the destination domain D The angle between them, P i P k and P D These represent the position vectors of control node i, adjacent node k, and destination domain D, respectively. The sign function sgn is used to determine whether the direction is clockwise or counterclockwise.

[0014] In some embodiments of the present invention, the deep reinforcement learning neural network is a deep Q-learning network. The model parameters that need to be set in the initial stage of training the deep Q-learning network include the decay value of future rewards, the learning rate of errors, and the random exploration rate. The state vector of the deep Q-learning network includes orientation awareness information, link distance, and link bandwidth. During the training process of the deep Q-learning network, the state information and orientation awareness information are used as inputs, and the next-hop node of the routing policy is used as the output.

[0015] In some embodiments of the present invention, the step of making cross-domain routing decisions using a pre-trained direction-aware assisted deep reinforcement learning neural network included in the inter-domain manager includes: inputting the latest collected direction-aware information and state information into the direction-aware assisted deep reinforcement learning neural network to obtain the next-hop node of the output cross-domain routing strategy.

[0016] In some embodiments of the present invention, the reward function of the direction-aware deep reinforcement learning neural network is defined in formula form as follows:

[0017]

[0018] Where α, β, and γ are weighting factors, v d It is the target agent, Ψ is the positive excitation constant, Φ is the negative penalty constant, and D i，k It is direction perception information. The link delay is obtained by dividing the link distance by the speed of light. i This represents the current agent, BWsel is the bandwidth used by the current path, and BWmax is the maximum link bandwidth.

[0019] Corresponding to the above methods, the present invention also provides a multi-agent hybrid routing system based on direction-aware assistance, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described in the above embodiments.

[0020] Corresponding to the above method, the application also provides a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the method in any one of the above embodiments.

[0021] Corresponding to the above method, the application also provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the method in any one of the above embodiments.

[0022] The multi-agent hybrid routing method based on direction perception assistance provided by the application can solve the problems of large communication overhead and poor adaptability to dynamic changes of satellite network topology of the existing centralized routing algorithm. On the other hand, the method can improve the problem of excessive dependence on local information in the distributed routing algorithm, which leads to the lack of global view when dealing with complex networks. The method skillfully combines the advantages of centralized and distributed routing, divides the network into multiple sub-domains, and each sub-domain is managed by a local control entity. The method can not only reduce the risk of single point failure, but also significantly improve the scalability and response speed of the network. In the LEO satellite network, the hybrid routing method can achieve a perfect balance between global optimization and local adaptability through sub-domain division and fine management of local control entities. The additional advantages, purposes, and features of the application will be partially described in the following description, and will become partially clear to those skilled in the art after studying the following part, or can be known according to the practice of the application. The purposes and other advantages of the application can be achieved and obtained by the structure specifically indicated in the specification and the drawings.

[0023] Those skilled in the art will understand that the purposes and advantages that can be achieved by the application are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the application will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings described herein are used to provide further understanding of the application, constitute a part of the application, and do not constitute a limitation of the application. In the drawings:

[0025] Figure 1 The flow chart of the multi-agent hybrid routing method based on direction perception assistance in an embodiment of the application.

[0026] Figure 2 The flow chart of the multi-agent hybrid routing method based on direction perception in another embodiment of the application.

[0027] Figure 3 The system block diagram of the multi-agent hybrid routing method based on direction perception in an embodiment of the application.

[0028] Figure 4 Convergence graph of the reward function of the DAMHR algorithm in an embodiment of the present application.

[0029] Figure 5 Schematic diagram of comparison between the method of the present application and the traditional method in terms of end-to-end delay index

[0030] Figure 6 Schematic diagram of comparison between the method of the present application and the traditional method in terms of bandwidth utilization index.

[0031] Figure 7 Structure diagram of a computer device carried in a multi-agent hybrid routing system in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not intended to limit the present application.

[0033] It should be further noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0034] It should be emphasized that the term “comprises / comprising” as used herein means the presence of the stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.

[0035] It should be further noted that, unless otherwise specified, the term “connected” as used herein can not only mean direct connection, but also indirect connection in the presence of an intermediate.

[0036] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0037] In order to overcome the problems existing in the prior art, the present application proposes a multi-agent hybrid routing method and system based on direction perception assistance. The method is applied to a low earth orbit satellite network environment, and direction perception assistance based on deep reinforcement learning is introduced. The (multi) agent trained based on the deep reinforcement learning model performs hybrid routing.

[0038] Deep Reinforcement Learning (DRL) is an advanced machine learning method that focuses on learning optimal decision-making strategies through continuous interactions between agents and environments. With its ability to handle highly complex problems and adapt to environmental changes, DRL has shown great potential in various fields, including routing optimization. In the specific field of satellite networks, given the significant increase in network dynamics and the increasing diversity of business needs, the introduction of Deep Q-Learning (DQN) algorithm as an important implementation method under the DRL framework has become increasingly important. DQN algorithm uses neural networks to approximate Q functions, enabling effective learning and optimization of routing strategies in complex and dynamic environments. This feature is particularly important for satellite networks, which contain a large number of satellite nodes and exhibit periodic dynamic changes in topology, making it extremely costly to fully observe network states. The Q function is a right-tail function of the standard normal distribution, and the update of the Q function enables the deep reinforcement learning model to learn the optimal strategy.

[0039] Figure 1 The flowchart of the Direction-Aware Multi-Agent Hybrid Routing (DAMHR) algorithm based on direction perception assistance in an embodiment of the present application. This method is implemented in a pre-built satellite network environment model. The satellite network is pre-divided into multiple sub-domains, each containing a core control satellite. The satellite is used as a node in the satellite network, and the core control satellite is pre-deployed with a domain manager and an inter-domain manager. The method includes the following steps:

[0040] Step S110: In the pre-built satellite network environment model, the domain manager is used to collect the state information within the domain, and the inter-domain manager is used to collect direction perception information and inter-domain state information from adjacent domain nodes through the flooding mechanism; wherein the state information includes multiple types of link bandwidth, link delay, link load, and network topology information.

[0041] In the specific implementation process, the domain manager is responsible for intra-domain routing processing, and the inter-domain manager is responsible for cross-domain routing processing. As shown in Figure 2 The domain manager and the inter-domain manager are pre-deployed in the core control satellite. The inter-domain manager includes a state extractor for obtaining state information and a pre-trained DRL neural network for making cross-domain routing decisions. The domain manager includes an SDN controller for centralized routing and a corresponding program based on the centralized routing algorithm for intra-domain routing decisions.

[0042] The state information is not limited to this, and the above listed are only examples.

[0043] Step S120: The core control satellite receives the data packet to be forwarded and the corresponding routing request; the data packet is from each satellite in the sub-domain or the core control satellite of other sub-domains.

[0044] Generally, the process of routing decision is that the source node receives a newly initiated routing request, and the source node sends the routing request to the core control satellite of the domain where the source node is located. The information of the routing request can include but is not limited to the source node, the target node, and the routing information category, etc.

[0045] Step S130: When the core control satellite analyzes the routing request and determines that the data packet is in the inter-domain routing stage, the cross-domain routing decision is made by using the pre-trained deep reinforcement learning neural network based on direction perception assistance contained in the inter-domain manager.

[0046] Step S140: When the core control satellite analyzes the routing request and determines that the data packet reaches the target domain, the intra-domain routing decision is made by using the domain manager deployed on the core control satellite according to the centralized routing method.

[0047] Wherein, reaching the target domain means entering the intra-domain routing stage. If the data packet is transmitted within the sub-domain from the beginning, it directly enters the intra-domain routing stage and adopts the centralized routing method for intra-domain routing decision.

[0048] By using this multi-agent hybrid routing method based on direction perception assistance, on the one hand, the problems of large communication overhead and poor adaptability to dynamic changes of satellite network topology of existing centralized routing algorithms can be solved. On the other hand, the problem that the global view is missing due to excessive dependence on local information when the distributed routing algorithm is used to process complex networks can be improved. This method skillfully combines the advantages of centralized and distributed routing, divides the network into multiple sub-domains, and each sub-domain is managed by a local control entity. This method can not only reduce the risk of single point failure, but also significantly improve the scalability and response speed of the network. In LEO satellite networks, the hybrid routing method can achieve a perfect balance between global optimization and local adaptability through sub-domain division and fine management of local control entities. The additional advantages, objects, and features of the present application will be partially set forth in the following description, and will become apparent to those skilled in the art after studying the following text, or can be learned from the practice of the present application. The objects and other advantages of the present application can be achieved by the structure specifically pointed out in the specification and the drawings.

[0049] In some embodiments of the present application, before making a routing decision, the method further comprises a preparation step of: building a satellite network environment model in advance for a low earth orbit satellite network, and generating a state space after the satellite network environment model is built, the state space containing state information within the domain, state information between the domains and the direction awareness information.

[0050] With the embodiments of the present application, the satellite network environment can be modeled, which lays the foundation for subsequent deep reinforcement learning-based routing methods.

[0051] In some embodiments of the present application, during the routing process, the method further comprises: for a newly arrived routing request, recording service request information, the service request information containing multiple types of source nodes, destination nodes, time durations and service requirement widths.

[0052] With the embodiments of the present application, the implementation process of hybrid routing in the satellite network environment is further clearly described.

[0053] The direction awareness information mentioned in step S110 of the present application can be obtained in various ways. On the one hand, based on the trajectory parameters and historical link quality data (link delay and link bandwidth) of the satellite, the approximate direction awareness information of the target domain can be derived, and the direction awareness information does not need to be completely accurate, and the approximate direction is sufficient as a decision reference. On the other hand, the direction awareness information can be defined by a formula.

[0054] In some embodiments of the present application, the direction awareness information is derived based on a position aided routing protocol through historical trajectory parameters and historical link quality data of the satellite; wherein the historical link quality data contains link delay and link bandwidth. Further, the link delay and link bandwidth can be weighted and averaged respectively, and the formula is as follows:

[0055]

[0056] In some other embodiments of the present application, the direction awareness information fuses the included angle and the direction information, and is defined in the formula as follows:

[0057] D i,k = cos(θ i , k)· sgn((P k -P i )×(P D -P i ));

[0058] Wherein, D i,k is the direction awareness information, θ i , k is the direction vector from the current control node i to the adjacent domain node k and the direction vector from the current control node i to the destination domain D the angle between P i , P k and P D respectively represent the position vectors of the control node i, the adjacent node k and the destination domain D, and the sign function sgn is used to determine whether the direction is clockwise or counterclockwise.

[0059] According to the embodiment of the application, the direction perception information can be obtained in various ways, and the direction perception information is used to provide a reference in the cross-domain routing decision.

[0060] In some embodiments of the application, the type of the deep reinforcement learning neural network is a deep Q learning network, and the model parameters to be set in the initial stage of training of the deep Q learning network include a decay value of future rewards, a learning rate of errors and a random exploration rate, and the state vector of the deep Q learning network includes direction perception information, link distance and link bandwidth.

[0061] Correspondingly, in the training process of the deep Q learning network, the state information and the direction perception information are used as inputs, and the next hop node of the routing strategy is used as output. Similarly, in the step of using the pre-trained deep reinforcement learning neural network based on direction perception assistance to make a cross-domain routing decision in step S130, the step includes: inputting the collected latest direction perception information and state information into the deep reinforcement learning neural network based on direction perception assistance to obtain the output next hop node of the cross-domain routing strategy.

[0062] According to the embodiment of the application, the deep Q learning network can be used to learn the correlation between the state vector including the direction perception information and the routing decision, and to learn the routing decision based on the state vector.

[0063] In some embodiments of the application, the reward function of the deep reinforcement learning neural network based on direction perception assistance is defined in the formula as follows:

[0064]

[0065] wherein, α, β and γ are weight factors, v d is a target agent, Ψ is a positive incentive constant, Φ is a negative punishment constant, D i，k is direction perception information, is a link delay obtained by dividing the link distance by the speed of light, a i represents a current intelligent agent, BWsel is the bandwidth used by the current path, and BWmax is the maximum link bandwidth.

[0066] Using this embodiment of the invention, a deep Q-learning network can be trained using the above activation function to learn the association between a state vector containing orientation-aware information and a routing decision, thereby enabling cross-domain routing decisions based on the trained deep Q-learning network.

[0067] In the specific implementation of the method proposed in this invention, it can be divided into two stages: (1) the preparation stage of the deep Q network model and agent for cross-domain routing decision; and (2) the stage of routing decision based on the deployed deep Q network model and agent.

[0068] First, the preparation phase of the deep Q-network model and agent used for cross-domain routing decisions is introduced, which includes the following 5 steps.

[0069] Step 1: Model the satellite network environment and generate the state space. The state space is a set of all possible states of a system.

[0070] For newly arriving services, record service request information: source node, destination node, duration, and service demand width, and collect network information: link bandwidth information, direction awareness information, and network topology information, together to generate the current state space;

[0071] The state vector of Agent i is defined as follows: Where D i，k It records the direction perception information of the current domain and adjacent domains. It is calculated by predicting the direction information Dir(A,D) of the target domain and the position information of adjacent domains through the current control node. The value is positive when it is close to the target domain and negative when it is far away. This represents the link transmission distance between the current domain d and the adjacent domain d'. This represents the link bandwidth between the current domain d and the adjacent domain d'.

[0072] Step 2: Design the action space. The action space is the set of all possible actions an agent can perform in reinforcement learning. In this method, the agent can be a core control satellite.

[0073] Action space of Agent i: Define the action space of agent i as Ai = Vi, where Vi is the set of neighboring agents of agent i. Therefore, for each agent, the size of the action space is the number of neighboring agents. Assume that data packet k arrives at the head of the receiving queue at time t, and agent i, according to the current state S... i，t and A i Select forwarding action S i，t .

[0074] Step 3: Design a reward mechanism to optimize the goal based on the status and actions.

[0075] Reward function: To optimize routing decisions in the direction-aware assisted multi-agent hybrid routing algorithm, the reward of Agenti is defined as:

[0076]

[0077] where α, β, γ are weight factors, v d is the target agent, Ψ is the positive incentive constant, Φ is the negative punishment constant. BWsel is the bandwidth used by the current path, BWmax is the maximum link bandwidth. D i，k is introduced in the reward function to drive routing. Direction-aware information D i，k is introduced in the reward function to drive routing. The reward function optimizes routing decisions by combining direction-aware information, propagation delay, and bandwidth utilization in an integrated manner. The positive reward of direction-aware information α·D i，k encourages agents to choose paths closer to the target; the negative reward of propagation delay encourages the selection of low-delay paths; the positive reward of bandwidth utilization improves bandwidth usage efficiency. The incentives Ψ and penalties Φ under specific conditions further optimize the decision-making of the agent under given background. The overall design ensures that the agent makes efficient and expected routing choices in complex network environments.

[0078] Step 4: Establish and train the double deep Q network model, and adjust the parameters of the model.

[0079] In order to more effectively capture the dynamic characteristics of satellite networks, a decentralized training and decentralized execution framework is adopted. The DQN network uses a neural network to approximate the state-action value function (Q-value function), and stores the experience of the agent (including state, action, reward, next state, etc.) through the experience replay buffer. The DQN model contains two neural networks with the same structure but different parameters: the target network (Qtar) and the evaluation network (Qeva). The target network is responsible for storing historical model parameters, plays a role in stabilizing the training process, and is updated regularly; the evaluation network evaluates the current state through the latest parameters and selects the optimal action for the newly arrived service request.

[0080] Step 5: Deploy the agent in the core satellite node and make routing decisions.

[0081] In LEO satellite networks, a hierarchical management architecture is adopted, dividing satellites into multiple domains and managing through core control satellites. Core control satellites monitor network status and topology in real time within the domain and exchange information with adjacent domains to achieve global situation control. The inter-domain manager contains a state extraction module, a deep reinforcement learning (DRL) neural network, and a routing decision module. The state extraction module collects key information from the network, such as network topology, link bandwidth, and target domain direction perception information, and inputs it into the DRL neural network to generate the optimal cross-domain path. The domain manager is responsible for intra-domain routing decisions, obtaining global data through centralized routing algorithms and controller interactions, and calculating the optimal intra-domain route to ensure efficient and accurate routing. To avoid local optimization problems, a location-aided routing protocol (LAR) is introduced based on direction perception technology. The core control satellite predicts the approximate direction of the target domain and integrates the direction information into the routing decision, effectively avoiding path length and loop problems, simplifying the state space, and improving routing decision efficiency.

[0082] On this basis, a multi-agent hybrid routing based on direction perception is realized. Specifically, when the routing request arrives at the source node, the core control satellite processes the intra-domain routing through the domain manager using the Dijkstra algorithm, and if the target node is in another domain, the inter-domain manager collects network state information and inputs it into the deep reinforcement learning (DRL) model combined with direction perception to calculate the cross-domain path. The agent transmits data hop by hop through distributed decision-making, and finally transmits to the target domain gateway, and then the domain manager optimizes the intra-domain routing to complete data transmission. This method combines distributed decision-making and centralized algorithms, achieving efficient and optimized cross-domain and intra-domain routing.

[0083] The phase of routing decision-making based on the deployed deep Q network model and agent is introduced. Figure 2 The flowchart of the multi-agent hybrid routing method based on direction perception in another embodiment of the present application is shown. First, the satellite network is modeled and state collection is performed, and when the routing request is initiated and initial decision-making is made, it is checked whether the target node is in the same domain. If yes, intra-domain routing decision-making is realized based on the Dijkstra algorithm, and if not, the cross-domain routing decision-making phase is entered. In the cross-domain routing decision-making phase, the inter-domain manager first collects state information, then performs cross-domain routing decision-making based on direction perception information, and transmits data packets hop by hop to the gateway node of the target domain. The target domain manager performs intra-domain routing decision-making. When everything is complete, the data is transmitted to the target node, and the routing is completed.

[0084] The phase of routing decision-making based on the deployed deep Q network model and agent contains the following 5 steps:

[0085] Step 1: Modeling of satellite network and state collection.

[0086] In the implementation of the present application, first of all, the LEO satellite network is modeled, considering a satellite network composed of multiple satellite domains D = {1, 2,..., 6}. For each domain d, define the set of satellite nodes N d = {1, 2,..., 16}, and establish the set of intra-domain links , the satellite links between domains Ensure that each satellite maintains contact with satellites in adjacent domains. In addition, a set of link distance parameters and a set of inter-domain link distance parameters are obtained by simulation in domain d: In order to manage these satellites, they are divided into 6 satellite domains, each containing 16 satellites. Each satellite is connected to its 4 neighboring satellites in the network (two on-orbit neighbors and two on adjacent orbit neighbors) through bidirectional links.

[0087] After modeling, the state space is defined The state vector includes the following information: direction-aware information D i,k and link distance and link bandwidth

[0088] In a satellite network, the position and direction information of a node is three-dimensional, and considering the wrap-around characteristics of the satellite network, the direction needs to be determined more accurately. In order to better reflect the direction-aware information, we need to consider not only the included angle, but also the explicit direction (clockwise or counterclockwise). D i,k It can be defined as:

[0089] D i，k = cos (θ i , k) · sgn ((P k -P i ) x (P D -P i ));

[0090] θ i , k is the direction vector from the current control node i to the adjacent domain node k and the included angle between the direction vector v → i, D from the current control node i to the destination domain D. P i , P k and PD represent the position vectors of control node i, adjacent node k and destination domain D respectively. The sign function sgn is used to determine whether the direction is clockwise or counterclockwise. D i，k The value of D i，kThe numerical value reflects the orientation from the controller node i to the adjacent node k and the direction to the destination domain. Regardless of the clockwise or counterclockwise direction, the D i，k The path with a larger absolute value will lead to a more efficient approach to the destination domain D.

[0091] Step 2: Initiation of routing request and initial decision.

[0092] In the initiation of routing request and initial decision phase, after the source node receives a new request, it sends it to the core control satellite of the domain it belongs to. As shown in Figure 3 The core control satellite, as the key node of routing management, is deployed with a domain manager and an inter-domain manager, responsible for intra-domain and cross-domain routing processing, respectively. Among them, Figure 3 The system block diagram of the multi-agent hybrid routing method based on direction perception in an embodiment of the present application. The left side is the entity architecture of the satellite network, the lower left side is the ground network, the upper left side is the LEO satellite network environment, and each dashed box represents each satellite in a sub-domain. The core control satellite is located in the center of each sub-domain, and the bidirectional link, inter-domain link, intra-domain link, and control link are shown in the figure. The right side is the domain manager and inter-domain manager deployed on the core control satellite, each domain manager contains an SDN controller and a centralized routing algorithm program for intra-domain routing decision-making using centralized method, and each inter-domain manager contains a state extractor for state information acquisition and a DRL neural network for cross-domain routing decision-making.

[0093] In specific implementation, the domain manager first collects the state information and traffic characteristics of the satellite network (such as link bandwidth, network load, etc.), ensuring the accuracy of routing calculation. Then, it determines whether the target node is in the same domain. If so, it uses a centralized algorithm (such as Dijkstra's algorithm) to calculate the optimal intra-domain path, meeting the quality of service (QoS) requirements and ensuring the efficiency of routing. If the target node is located in other domains, the inter-domain manager takes over the task and starts the cross-domain routing mechanism. This initial judgment step is crucial as it quickly determines the routing strategy and ensures the efficiency and flexibility of data transmission. This example selects the case where the target node is not in the same domain as the source node for introduction.

[0094] Step 3: State information collection by inter-domain manager.

[0095] First, the inter-domain manager collects direction perception information from adjacent inter-domain nodes through the flooding mechanism, including network state feedback such as link bandwidth, delay, and load. The state information after flooding will be used to determine the general direction of the domain where the target node is located. In order to reduce the consumption of network resources caused by frequent flooding, subsequent routing requests can be derived based on historical data obtained from previous flooding.

[0096] Assuming the orbit of the target domain in the satellite network is known, combined with the satellite's trajectory parameters and historical link quality data, the relative position change of the target domain can be predicted. This derivation process can be done by time-weighted average method. Let the link delay in the flooding result be D i , and the bandwidth be B i , where i represents the i-th flooding, and the flooding result closest to the current time is given a higher weight.

[0097] The weighted average calculation formula of direction perception information is:

[0098]

[0099] Where ω i is the weight of the i-th flooding, usually set in a time-decreasing manner, and the flooding result closer to the current time has a higher weight. The weighted average link delay and bandwidth calculated can derive the relative direction of the domain where the target node is located.

[0100] Step 4: DRL agent makes cross-domain routing decision.

[0101] A double deep Q network model is established and trained, with the decay value of future reward γ set to 0.99, the learning rate of error α set to 0.0001, and the random exploration rate ε set to 0.1.

[0102] Based on the collected direction perception information, the inter-domain manager inputs these network state data into the deep reinforcement learning (DRL) model. According to the real-time network state and direction information, the DRL model selects the optimal next-hop node through the decision strategy derived by training. At each selection, the agent will give priority to those nodes with better network conditions and towards the target domain direction, to ensure the data packet moves towards the target domain step by step. When the data packet is forwarded hop by hop to the gateway node of the target domain, it enters the next step.

[0103] The inter-domain manager combines the collected direction perception information with the trained agent decision strategy to select the optimal next-hop node. The agent uses deep reinforcement learning (DRL) algorithm, which not only relies on real-time network conditions, but also accumulates knowledge of historical link conditions through continuous learning, enabling it to predict future link changes. This decision-making process will prioritize nodes with good network performance (such as high bandwidth, low latency) and towards the target domain direction, ensuring that the data packet approaches the target domain step by step at each hop. This distributed routing decision mechanism, by utilizing inter-domain agents, makes cross-domain routing more flexible and adaptive to different network conditions and topology changes.

[0104] Step 5: The target domain manager performs intra-domain routing decision.

[0105] Data packets are forwarded hop by hop, and finally reach the gateway node of the target domain, completing the inter-domain routing task. At this time, the domain-internal routing decision phase is entered. The core control satellite of the target domain takes over the data packets and the domain manager deployed inside it is responsible for executing the domain-internal routing decision. The specific implementation details are as follows: First, the domain manager will immediately collect the network state information in the current target domain, including link bandwidth, delay, load condition, etc., to ensure a comprehensive grasp of the network topology and link quality. Next, the domain manager performs intra-domain path computation using a centralized algorithm (such as Dijkstra's algorithm) based on the location of the target node and network state. Dijkstra's algorithm can quickly find the optimal path from the gateway node to the target node based on the weight of each link (such as bandwidth, delay, etc.), ensuring that the path meets the Quality of Service (QoS) requirements.

[0106] Once the data packets reach the gateway node of the target domain, the system enters the intra-domain routing phase. In this phase, the routing mechanism switches from distributed to centralized. The core control satellite of the target domain immediately acquires and updates the network state information in the target domain, such as link bandwidth, delay, load, etc., through the domain manager deployed inside it. By comprehensively grasping these real-time information, the domain manager can quickly make intra-domain routing decisions. Unlike inter-domain routing, intra-domain routing uses traditional centralized algorithms (such as Dijkstra's algorithm), which can quickly find the optimal path from the gateway node to the target node based on the weight of each link (such as bandwidth, delay, etc.), ensuring that the path meets the Quality of Service (QoS) requirements.

[0107] During the calculation process, the domain manager will consider the real-time state of the link and preferentially select links with sufficient bandwidth, low delay, and balanced load to ensure efficient transmission of data packets to the target node. Specifically, Dijkstra's algorithm calculates the shortest path P(s, t) from the gateway node s to the target node t based on the link weight ω(e) in the network graph, ensuring that the selected path P satisfies the optimality:

[0108] P(s, t) = min∑ e∈P ω(e);

[0109] Through this path calculation, the domain manager finally determines an optimal transmission path and transmits the data packets hop by hop along this path to the target node. Once the data packets successfully reach the target node, the entire routing process ends and the intra-domain routing decision is successfully completed.

[0110] In the training process of the intelligent agent, the direction perception information based on the location-aided routing protocol (LAR) is introduced. By integrating this information into the reward function of the intelligent agent, the local optimal solution caused by local observation and the non-stationary problem caused by frequent changes in network topology can be effectively avoided, thereby improving the stability and reliability of the overall routing. Through this method, the adaptability and global optimization ability of the routing decision in a dynamic environment can be improved under the premise of ensuring low communication overhead, and the method is suitable for complex satellite network communication environments.

[0111] The multi-agent hybrid routing method proposed in the application can combine the intelligent routing mechanism of centralized and distributed mechanisms to effectively cope with complex routing requirements in large-scale satellite networks. By intelligently combining these two routing mechanisms, the flexibility of the distributed mechanism and the global optimization ability of the centralized mechanism are fully utilized, enabling the entire network to efficiently and dynamically handle cross-domain and intra-domain routing tasks.

[0112] On the one hand, in the cross-domain routing process, the system realizes distributed intelligent routing decision-making through a deep reinforcement learning (DRL) model. Each inter-domain manager acts as an independent agent and selects the optimal next-hop node through the DRL model based on collected network state data, direction perception information, and other inputs. This distributed decision-making approach can respond in real-time to changes in network state, including link bandwidth, latency, load, etc., ensuring that data packets move gradually in the direction of the target domain and avoiding single-point failure problems that may exist in traditional centralized routing. At the same time, the DRL model can effectively accumulate historical data through continuous training and learning, thereby predicting future link conditions and further improving the accuracy and reliability of decision-making.

[0113] On the other hand, when the data packet reaches the gateway node of the target domain, the system switches to the centralized routing mechanism. The core control satellite of the target domain collects comprehensive network state information within the domain through the domain manager, including link bandwidth, latency, load, etc., and uses Dijkstra and other classic algorithms for global optimization calculation of intra-domain paths. Compared to distributed decision-making, centralized algorithms can quickly find the optimal path from the gateway node to the target node under known network topology, ensuring that the path meets quality of service (QoS) requirements such as low latency, high bandwidth, and load balancing. This centralized routing decision effectively utilizes the global view of the network and can ensure optimal path selection within a small-scale topology.

[0114] Finally, the effectiveness of the method is verified through experiments and compared with existing technologies.

[0115] The experimental conditions were as follows: Under the environment of Intel(R)Xeon(R)Gold 5218R CPU@2.10GHz, we used the MATLAB AERO toolbox in Matlab2022 to simulate a Walker constellation and used the reinforcement learning toolbox to train the agent.

[0116] In this invention, we designed and compared three routing algorithms covering distributed intelligent routing and centralized routing paradigms. These include the centralized shortest path routing algorithm (SR-C), the distributed shortest path routing algorithm (SR-D), and the reinforcement learning-based distributed routing algorithm Q-routing. Experimental results are as follows:

[0117] (1) As Figure 4 As shown, Figure 4 This is a convergence graph of the reward function of the DAMHR algorithm in one embodiment of the present invention. The real part is the result of averaging the shaded area every ten iterations. As can be seen from the graph, the DAMHR algorithm converges quickly with minimal fluctuations. After the algorithm reached convergence, its latency and bandwidth utilization were evaluated.

[0118] (2) Figure 5 The end-to-end latency performance of four algorithms under different loads is demonstrated. Figure 5 This diagram illustrates a comparison of the end-to-end latency metrics between the method of this invention and conventional methods. As the load increases, the DAMHR algorithm exhibits significantly lower latency than other algorithms and demonstrates higher robustness, with its latency decreasing rapidly and remaining stable. In contrast, the SR-C algorithm experiences a sharp increase in latency when the load exceeds 300 Mbps, while the SR-D algorithm shows a slower but limited increase in latency, particularly under high load conditions. Conversely, the Q-routing algorithm exhibits consistently high latency across the entire load range, especially under high loads.

[0119] (3) Figure 6 This demonstrates how bandwidth utilization varies with load. The DAMHR algorithm significantly improves bandwidth utilization and maintains high stability under various load conditions. Figure 6 This diagram illustrates a comparison of the bandwidth utilization metrics between the method of this invention and traditional methods. In contrast, the SR-C algorithm shows a significant decrease in bandwidth utilization under high load conditions, while the SR-D algorithm exhibits a slight decrease in bandwidth utilization as the load gradually increases. The Q-routing algorithm maintains a consistently low level of bandwidth utilization. Therefore, the DAMHR algorithm demonstrates superior performance and greater robustness in terms of both latency and resource utilization.

[0120] Corresponding to the above method, the application also provides a multi-agent hybrid routing system based on direction perception assistance, which comprises a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, so that the system implements the steps of the method as described above. Wherein, the system is mounted on the computer device of the core control satellite.

[0121] In summary, the multi-agent hybrid routing method based on direction perception proposed in the application has the following key points and advantages:

[0122] (1) By using the proposed domain-layered routing calculation strategy, the routing task is divided into intra-domain and inter-domain parts, which are processed by different algorithms. Among them, the inter-domain routing uses agents trained based on deep reinforcement learning (DRL) for distributed routing decision, and the intra-domain routing uses centralized routing decision, so as to improve the calculation efficiency and the accuracy of path selection.

[0123] (2) In the training process of the agent, by introducing the direction perception information based on the spatial position, the adaptability and efficiency of the routing decision in the dynamic network environment can be improved. By integrating the direction perception information into the reward function of the agent, the local optimal solution caused by local observation and the non-stationary problem caused by frequent changes in network topology can be effectively avoided, so as to improve the stability and reliability of the overall routing.

[0124] (3) By using a new routing mechanism combining centralized and distributed methods, the intra-domain routing and inter-domain routing tasks are ingeniously separated, and optimized routing strategies are designed for both. In the intra-domain, a centralized routing algorithm is used to improve the calculation efficiency and ensure fast and accurate path selection in a small range. In the inter-domain, a distributed routing algorithm based on deep reinforcement learning (DRL) is used to enable the agent to dynamically adapt to the frequent changes in satellite network topology. This combined strategy ensures the effective integration of global view and local information, avoids the limitations of a single algorithm, improves the overall adaptability and flexibility of the network, and reduces the communication overhead.

[0125] (4) By introducing the direction perception auxiliary multi-agent routing algorithm, the direction perception information of the spatial position is integrated into the training process of the agent, which can enable the agent to make more adaptive routing decisions in a complex dynamic satellite network environment. The direction perception information provides more detailed topology structure perception, enabling the agent to identify potential directional transmission requirements in the network and effectively avoid the generation of local optimal solutions. In addition, the introduction of this information enhances the robustness of the system in response to frequent changes in topology, improves the stability of the algorithm and the overall efficiency of network transmission, and especially in a dynamic environment, better adapts to changes and achieves more optimal path planning.

[0126] (5) Through the global perspective of the multi-agent system, the load balancing capability of the satellite network is significantly improved, and the problem of uneven resource allocation caused by insufficient local information in traditional routing algorithms is solved. The agent can monitor the traffic distribution, path load and bandwidth usage of the entire network in real time, make dynamic decisions based on the global perspective, and effectively avoid the overload problem of hot paths. When the agent discovers that the traffic of some links is too large, it will intelligently disperse the traffic to lightly loaded paths, thereby reducing the probability of communication bottlenecks. This global optimization strategy not only improves the overall throughput of the network, but also improves the stability and communication efficiency of the system in a complex dynamic environment. At the same time, the agent has adaptive ability and can optimize resource allocation in real time according to network load changes, ensuring low delay and high bandwidth utilization in high load or burst conditions, greatly enhancing the robustness and adaptability of the network.

[0127] (6) The above, the method proposed in the application can improve the adaptability and global optimization capability of routing decisions in a dynamic environment while ensuring low communication overhead, and is suitable for complex satellite network communication environments.

[0128] The embodiment of the application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0129] Referring to Figure 7 , Figure 7Fig. 1 is a schematic diagram of a computer device 00 according to an embodiment of the present application. The computer device 00 comprises a processor 01, a memory 02, and a computer program stored in the memory 02 and executable by the processor 01. The processor 01 implements the human data server access control method according to the above-mentioned method embodiments when executing the computer program.

[0130] The processor 01 and the memory 02 are connected, for example, through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor 01 can implement or execute various exemplary logical blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor 01 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc. The bus 03 can include a path for transmitting information between the above-mentioned components. The bus 03 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The memory 02 is used to store a computer program corresponding to the human data server access control method according to the above-mentioned embodiments of the present application. The computer program is executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to realize the content shown in the above-mentioned method embodiments.

[0131] Corresponding to the above method, the present application also provides a computer readable storage medium, which stores computer program / instructions, and the computer program / instructions are executed by a processor to realize the steps of the method according to any one of the above embodiments. The computer readable storage medium can be a tangible storage medium, such as random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable memory disk, CD-ROM, or any other form of storage medium known in the art.

[0132] Corresponding to the above method, the present application also provides a computer program product, which comprises computer program / instructions, and the computer program / instructions are executed by a processor to realize the steps of the method according to any one of the above embodiments.

[0133] It should be apparent to one of ordinary skill in the art that the example components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination thereof. The choice of hardware or software implementation is dependent on the particular application and design constraints imposed on the solution. Skilled artisans can employ a variety of approaches to implement the described functionality, and the choice of available approaches will be apparent to artisans from the disclosure herein. When implemented in hardware, the solution can be implemented as, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are the program or code segments to perform the necessary tasks. The program or code segments can be stored in a machine readable medium, or transmitted by a carrier wave over a transmission medium or a communication link.

[0134] It should be apparent to one of ordinary skill in the art that the example components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination thereof. The choice of hardware or software implementation is dependent on the particular application and design constraints imposed on the solution. Skilled artisans can employ a variety of approaches to implement the described functionality, and the choice of available approaches will be apparent to artisans from the disclosure herein. When implemented in hardware, the solution can be implemented as, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are the program or code segments to perform the necessary tasks. The program or code segments can be stored in a machine readable medium, or transmitted by a carrier wave over a transmission medium or a communication link.

[0135] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0136] The above merely illustrates the preferred embodiments of the present application, and is not used to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of protection of the present application.

Claims

1. A multi-agent hybrid routing method based on direction-aware assistance, characterized in that, The satellite network is pre-divided into multiple subdomains, each containing a core control satellite. The satellites serve as nodes in the satellite network, and each core control satellite is pre-deployed with a domain manager and an inter-domain manager. The method includes: In the pre-built satellite network environment model, the domain manager is used to collect state information within the domain, and the inter-domain manager is used to collect direction-aware information and inter-domain state information from neighboring inter-domain nodes through a flooding mechanism; wherein, the state information includes multiple types of link bandwidth, link latency, link load and network topology information; Direction-aware information, which integrates angle and direction information, is defined by the following formula: in, It is direction perception information. From the current control node To neighboring domain nodes Direction vector With the current control node To the destination domain Direction vector The angle between them , and Representing control nodes Adjacent nodes and target domain Position vector, sign function Used to determine whether the direction is clockwise or counterclockwise; The core control satellite receives data packets to be forwarded and corresponding routing requests; the data packets originate from satellites within the subdomain or from core control satellites in other subdomains. When the core control satellite parses the routing request and determines that the data packet is in the inter-domain routing stage, it uses the pre-trained direction-aware-assisted deep reinforcement learning neural network contained in the inter-domain manager to make cross-domain routing decisions; wherein, the direction-aware-assisted deep reinforcement learning neural network refers to a deep reinforcement learning neural network that integrates direction-aware information and state information in an integrated manner in the reward function to optimize the routing decision. Once the core control satellite resolves the routing request and determines that the data packet has arrived at the target domain, it uses the domain manager deployed on the core control satellite to make intra-domain routing decisions according to a centralized routing method.

2. The method according to claim 1, characterized in that, The method further includes: Before making routing decisions, a satellite network environment model is built for the low Earth orbit satellite network in advance, and a state space is generated after the satellite network environment model is built. The state space includes the state information within the domain, the state information between the domains, and the direction perception information. During the routing process, for each newly arrived routing request, service request information is recorded. This service request information includes multiple parameters such as the source node, destination node, duration, and service requirement width.

3. The method according to claim 1, characterized in that, The deep reinforcement learning neural network is a deep Q-learning network. The model parameters that need to be set in the initial stage of training the deep Q-learning network include the decay value of future rewards, the learning rate of errors, and the random exploration rate. The state vector of the deep Q-learning network includes orientation awareness information, link distance, and link bandwidth. During the training process of the deep Q-learning network, state information and orientation awareness information are used as inputs, and the next-hop node of the routing policy is used as the output.

4. The method according to claim 3, characterized in that, The step of making cross-domain routing decisions using a pre-trained direction-aware assisted deep reinforcement learning neural network included in the inter-domain manager includes: The latest direction-aware information and state information are input into the direction-aware-assisted deep reinforcement learning neural network to obtain the next-hop node of the output cross-domain routing strategy.

5. The method according to claim 3, characterized in that, The reward function of the direction-aware deep reinforcement learning neural network is defined in formula form as follows: in, , and It is a weighting factor. It is the target intelligent agent. It is a positive excitation constant. It is direction perception information. The link delay is obtained by dividing the link distance by the speed of light. This represents the current intelligent agent. This is the bandwidth used by the current path. It is the maximum link bandwidth.

6. A multi-agent hybrid routing system based on direction-aware assistance, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

Images