Multi-agent hybrid routing method and system based on direction perception assistance

Abstract

The invention provides a multi-agent hybrid routing method and system based on direction perception assistance, and the method comprises the steps: collecting the state information in a domain through a domain manager in a pre-built satellite network environment model, utilizing the inter-domain manager to collect direction sensing information and inter-domain state information from adjacent inter-domain nodes through a flooding mechanism; the core control satellite receives a data packet needing to be forwarded and a corresponding routing request; the data packets are from satellites in a sub-domain or core control satellites in other sub-domains; in the inter-domain routing stage, a deep reinforcement learning neural network is utilized to carry out cross-domain routing decision, and when a data packet arrives at a target domain, a centralized routing method is utilized to carry out intra-domain routing decision, in a reward function of a deep reinforcement learning neural network based on direction perception assistance, direction perception information and state information are fused in an integrated mode to optimize a routing decision. According to the method, the adaptability and the global optimization capability in a complex satellite network communication environment can be improved.

Description

Technical Field

[0001] The present invention relates to the field of satellite networks and industrial Internet technologies, and in particular to a multi-agent hybrid routing method and system based on direction perception assistance. Background Art

[0002] As global communication demand continues to surge, the deep integration of low earth orbit (LEO) satellite networks and existing ground networks has provided unprecedented opportunities for expanding communication coverage for the sixth generation mobile communication (6G) network. However, the high dynamics of LEO satellite nodes poses severe challenges to the addressing and routing strategies of large-scale constellations. Among them, in the construction of a large constellation network in a low earth orbit environment, the addressing and routing strategy refers to: ① determining the address of each satellite; ② determining the transmission path of the data packet in the network.

[0003] In the prior art, routing technology solutions to address the above challenges include:

[0004] (1) The centralized routing method relies on the software-defined network (SDN) controller as the core hub to achieve effective separation of the control plane and the data plane. However, as the scale of the network becomes larger and larger, the management complexity and resource overhead associated with flow_mod message transmission have increased significantly, becoming a major bottleneck restricting its application.

[0005] (2) The distributed routing method is that each satellite node independently collects surrounding network information and makes routing decisions based on it. Although this method gives the network a high degree of flexibility, over-reliance on local information may cause routing decisions to deviate from the global optimal solution, thereby creating an irreconcilable contradiction between the acquisition of network status information and the management granularity.

[0006] Therefore, how to provide a routing method that can overcome centralized routing methods and distributed routing methods and improve adaptability and global optimization capabilities in complex satellite network communication environments is a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a multi-agent hybrid routing method and system based on direction perception assistance to eliminate or improve one or more defects existing in the prior art.

[0008] One aspect of the present invention provides a multi-agent hybrid routing method based on direction perception assistance, wherein a satellite network is pre-divided into multiple sub-domains, each sub-domain includes a core control satellite, and the satellite is used as a node of the satellite network. A domain manager and an inter-domain manager are pre-deployed on the core control satellite, respectively. The method comprises: in a pre-built satellite network environment model, using the domain manager to collect status information within the domain, and using the inter-domain manager to collect direction perception information and inter-domain status information from adjacent inter-domain nodes through a flooding mechanism; wherein the status information includes multiple types of link bandwidth, link delay, link load and network topology information; the core control satellite receives a message to be forwarded. data packets and corresponding routing requests; the data packets come from satellites in the subdomain or core control satellites of other subdomains; when the core control satellite parses the routing request and determines that the data packet is in the inter-domain routing stage, the pre-trained deep reinforcement learning neural network based on direction perception assistance contained in the inter-domain manager is used to make cross-domain routing decisions; when the core control satellite parses the routing request and determines that the data packet arrives at the target domain, the domain manager deployed on the core control satellite is used to make intra-domain routing decisions according to the centralized routing method; wherein, the reward function of the deep reinforcement learning neural network based on direction perception assistance integrates direction perception information and state information in an integrated manner to optimize routing decisions.

[0009] In some embodiments of the present invention, the method also includes: before making a routing decision, building a satellite network environment model for the low-earth orbit satellite network in advance, and generating a state space after the satellite network environment model is built, the state space including the state information within the domain, the state information between the domains and the direction perception information; during the routing process, for newly arrived routing requests, recording service request information, the service request information including multiple sources, destination nodes, durations and service demand widths.

[0010] In some embodiments of the present invention, the direction perception information is derived based on a location-assisted routing protocol through historical trajectory parameters and historical link quality data of the satellite; wherein the historical link quality data includes link delay and link bandwidth.

[0011] In some embodiments of the present invention, the direction perception information is integrated with the angle and direction information, and is defined in the form of a formula as follows:

[0012] D i，k =cos(θ i , k)·sgn((P k -P i )×(P D -P i ));

[0013] Among them, D i，kis the direction perception information, θ i , k is the direction vector from the current control node i to the adjacent domain node k and the direction vector from the current control node i to the destination domain D The angle between i , P k and P D They represent the position vectors of the control node i, the adjacent node k and the destination domain D respectively. The sign function sgn is used to determine whether the direction is clockwise or counterclockwise.

[0014] In some embodiments of the present invention, the type of the deep reinforcement learning neural network is a deep Q learning network. The model parameters that need to be set in the initial stage of training the deep Q learning network include a decay value for future rewards, a learning rate for errors, and a random exploration rate. The state vector of the deep Q learning network includes direction perception information, link distance, and link bandwidth. During the training process of the deep Q learning network, the state information and direction perception information are used as input, and the next hop node of the routing strategy is used as output.

[0015] In some embodiments of the present invention, the step of using the pre-trained direction-awareness-assisted deep reinforcement learning neural network contained in the inter-domain manager to make cross-domain routing decisions includes: inputting the latest collected direction-awareness information and status information into the direction-awareness-assisted deep reinforcement learning neural network to obtain the output next-hop node of the cross-domain routing strategy.

[0016] In some embodiments of the present invention, the reward function of the deep reinforcement learning neural network assisted by direction perception is defined in formula form as follows:

[0017]

[0018] Among them, α, β and γ are weight factors, v d is the target agent, Ψ is the positive incentive constant, Φ is the negative penalty constant, D i，k is the direction perception information, is the link delay obtained by dividing the link distance by the speed of light, a i represents the current agent, BWsel is the bandwidth used by the current path, and BWmax is the maximum link bandwidth.

[0019] Corresponding to the above method, the present invention also provides a multi-agent hybrid routing system based on direction perception assistance, including a processor, a memory and a computer program / instructions stored in the memory, the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described in any one of the above embodiments.

[0020] Corresponding to the above method, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0021] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0022] The multi-agent hybrid routing method based on direction perception assistance proposed in the present invention can, on the one hand, solve the problems of large communication overhead and poor adaptability to dynamic changes in satellite network topology in existing centralized routing algorithms. On the other hand, it can improve the problem of the loss of global view caused by excessive reliance on local information when dealing with complex networks in distributed routing algorithms. The method cleverly combines the advantages of centralized and distributed routing, divides the network into multiple subdomains, and each subdomain is managed by a local control entity. The method can not only reduce the risk of single point failures, but also significantly improve the scalability and response speed of the network. In the LEO satellite network, the hybrid routing method can achieve a perfect balance between global optimization and local adaptability through the fine management of subdomain division and local control entities. The additional advantages, purposes, and features of the present invention will be partially explained in the following description, and will become partially obvious to ordinary technicians in the field after studying the following, or can be known from the practice of the present invention. The purposes and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the specification and the drawings.

[0023] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0025] Figure 1 This is a flow chart of a multi-agent hybrid routing method based on direction perception assistance in one embodiment of the present invention.

[0026] Figure 2 It is a flowchart of a multi-agent hybrid routing method based on direction perception in another embodiment of the present invention.

[0027] Figure 3 This is a system block diagram of a multi-agent hybrid routing method based on direction perception in one embodiment of the present invention.

[0028] Figure 4 This is a convergence diagram of the DAMHR algorithm reward function in one embodiment of the present invention.

[0029] Figure 5 Schematic diagram showing the comparison between the method of the present invention and the traditional method in terms of end-to-end delay index

[0030] Figure 6 The figure is a comparison diagram of the bandwidth utilization index between the method of the present invention and the traditional method.

[0031] Figure 7 It is a schematic diagram of the structure of a computer device installed in a multi-agent hybrid routing system in one embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0033] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0034] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0035] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0036] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0037] In order to overcome the problems existing in the prior art, the present invention proposes a multi-agent hybrid routing method and system based on direction perception assistance. The method is applied to a low-Earth orbit satellite network environment, introduces direction perception assistance based on deep reinforcement learning, and performs hybrid routing based on (multiple) agents trained based on deep reinforcement learning models.

[0038] Among them, Deep Reinforcement Learning (DRL) is an advanced machine learning method, the core of which is to learn the optimal decision-making strategy through continuous interaction between the agent and the environment. With its ability to handle highly complex problems and flexibly adapt to environmental changes, deep reinforcement learning has shown wide application potential in many fields, including routing optimization. In the specific field of satellite networks, in view of the significant enhancement of network dynamics and the increasing diversification of business needs, the Deep Q-learning (Deep Q-Network, DQN) algorithm has been further introduced as an important implementation method under the DRL framework. The DQN algorithm uses neural networks to approximate the Q function, so that it can effectively learn and optimize routing strategies in complex and changing environments. This feature is particularly important for satellite networks, because the network not only contains a large number of satellite nodes, but also its topological structure presents periodic dynamic changes, which makes the cost of fully observing the network status extremely high. Among them, the Q function is the right tail function of the standard normal distribution. The update of the Q function enables the deep reinforcement learning model to learn the optimal strategy.

[0039] Figure 1 The flowchart of the Direction-Aware Multi-Agent Hybrid Routing (DAMHR algorithm) method based on direction-aware assistance in one embodiment of the present invention is shown in FIG. The method is implemented in a pre-built satellite network environment model. The satellite network is pre-divided into multiple subdomains. Each subdomain includes a core control satellite. The satellite is used as a node of the satellite network. The core control satellite is pre-deployed with a domain manager and an inter-domain manager. The method includes the following steps:

[0040] Step S110: In the pre-built satellite network environment model, the domain manager is used to collect status information within the domain, and the inter-domain manager is used to collect direction perception information and inter-domain status information from adjacent inter-domain nodes through a flooding mechanism; wherein the status information includes multiple types of link bandwidth, link delay, link load and network topology information.

[0041] In the specific implementation process, the domain manager is responsible for routing within the domain, and the inter-domain manager is responsible for routing across domains. Figure 2 As shown, the domain manager and inter-domain manager are pre-deployed in the core control satellite. The inter-domain manager contains a state extractor for obtaining state information and a pre-trained DRL neural network for making cross-domain routing decisions. The domain manager contains an SDN controller for centralized routing and a corresponding program based on a centralized routing algorithm to make intra-domain routing decisions.

[0042] The status information is not limited to these, and the above listed ones are only examples.

[0043] Step S120: The core control satellite receives the data packet to be forwarded and the corresponding routing request; the data packet comes from each satellite in the subdomain or the core control satellite of other subdomains.

[0044] Generally speaking, the process of routing decision is: the source node receives a newly initiated routing request, and the source node sends the routing request to the core control satellite of the domain where the source node is located. The information of the routing request may include but is not limited to the source node, the target node and the routing information category.

[0045] Step S130: When the core control satellite parses the routing request and determines that the data packet is in the inter-domain routing stage, the pre-trained deep reinforcement learning neural network based on direction perception assistance contained in the inter-domain manager is used to make cross-domain routing decisions.

[0046] Step S140: When the core control satellite parses the routing request and determines that the data packet arrives at the target domain, the domain manager deployed on the core control satellite is used to make intra-domain routing decisions according to a centralized routing method.

[0047] Among them, arriving at the target domain, that is, entering the intra-domain routing stage, if the data packet is transmitted within the sub-domain from the beginning, it directly enters the intra-domain routing stage and uses a centralized routing method to make intra-domain routing decisions.

[0048] The use of this multi-agent hybrid routing method based on direction perception assistance can, on the one hand, solve the problems of large communication overhead and poor adaptability to dynamic changes in satellite network topology in existing centralized routing algorithms. On the other hand, it can improve the problem of the lack of global view caused by excessive reliance on local information when dealing with complex networks in distributed routing algorithms. This method cleverly combines the advantages of centralized and distributed routing, divides the network into multiple subdomains, and each subdomain is managed by a local control entity. This method can not only reduce the risk of single point failures, but also significantly improve the scalability and response speed of the network. In the LEO satellite network, the hybrid routing method can achieve a perfect balance between global optimization and local adaptability through the fine management of subdomain division and local control entities. Additional advantages, purposes, and features of the present invention will be partially explained in the following description, and will become partially obvious to ordinary technicians in the field after studying the following, or can be known from the practice of the present invention. The purposes and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the specification and the accompanying drawings.

[0049] In some embodiments of the present invention, before making a routing decision, the method also includes a preparation step: pre-building a satellite network environment model for the low-Earth orbit satellite network, and generating a state space after the satellite network environment model is built, wherein the state space includes the state information within the domain, the state information between the domains, and the direction perception information.

[0050] By adopting the embodiment of the invention, the satellite network environment can be modeled, laying the foundation for the subsequent routing method based on deep reinforcement learning.

[0051] In some embodiments of the present invention, during the routing process, the method further comprises: recording service request information for a newly arrived routing request, wherein the service request information includes multiple types of a source node, a destination node, a duration, and a service demand width.

[0052] The embodiment of the invention is used to further clearly illustrate the implementation process of hybrid routing in a satellite network environment.

[0053] The direction perception information mentioned in step S110 of the present invention can be obtained in a variety of ways. On the one hand, based on the trajectory parameters of the satellite and the historical link quality data (link delay and link bandwidth), the approximate direction perception information of the target domain can be derived. The direction perception information does not need to be completely accurate, and the approximate direction is sufficient as a reference for decision making. On the other hand, the direction perception information can be limited by a formula.

[0054] In some embodiments of the present invention, the historical trajectory parameters and historical link quality data of the satellite are derived based on the location-assisted routing protocol; wherein the historical link quality data includes link delay and link bandwidth. Further, the link delay and link bandwidth can be weighted averaged respectively, and the formula is as follows:

[0055]

[0056] In other embodiments of the present invention, the direction perception information is integrated with the angle and direction information and is defined in the form of a formula as follows:

[0057] D i,k =coS(θ i , k)·sgn((P k -P i )×(P D -P i ));

[0058] Among them, D i,k is the direction perception information, θ i , k is the direction vector from the current control node i to the adjacent domain node k and the direction vector from the current control node i to the destination domain D The angle between i , P k and P D They represent the position vectors of the control node i, the adjacent node k and the destination domain D respectively. The sign function sgn is used to determine whether the direction is clockwise or counterclockwise.

[0059] By adopting the embodiment of the present invention, direction perception information can be obtained in a variety of ways, and the direction perception information can be used to provide a reference in cross-domain routing decisions.

[0060] In some embodiments of the present invention, the type of the deep reinforcement learning neural network is a deep Q learning network. The model parameters that need to be set in the initial stage of training the deep Q learning network include a decay value for future rewards, a learning rate for errors, and a random exploration rate. The state vector of the deep Q learning network includes direction perception information, link distance, and link bandwidth.

[0061] Accordingly, in the training process of the deep Q learning network, the state information and the direction perception information are used as input, and the next hop node of the routing strategy is used as output. Similarly, in the step S130 of making cross-domain routing decisions using the pre-trained deep reinforcement learning neural network based on direction perception assistance contained in the inter-domain manager, it includes: inputting the latest direction perception information and state information collected into the deep reinforcement learning neural network based on direction perception assistance to obtain the output next hop node of the cross-domain routing strategy.

[0062] By adopting this embodiment of the invention, a deep Q learning network can be used to learn the association between a state vector containing direction perception information and a routing decision, and learn to make a routing decision based on the state vector.

[0063] In some embodiments of the present invention, the reward function of the deep reinforcement learning neural network assisted by direction perception is defined in the form of a formula as follows:

[0064]

[0065] Among them, α, β and γ are weight factors, v d is the target agent, Ψ is the positive incentive constant, Φ is the negative penalty constant, D i，k is the direction perception information, is the link delay obtained by dividing the link distance by the speed of light, a i represents the current agent, BWsel is the bandwidth used by the current path, and BWmax is the maximum link bandwidth.

[0066] By adopting this embodiment of the invention, the above-mentioned excitation function can be used to train a deep Q-learning network to learn the association between the state vector containing direction perception information and the routing decision, thereby making cross-domain routing decisions based on the trained deep Q-learning network.

[0067] The specific implementation process of the method proposed in the present invention can be divided into two stages: (1) the preparation stage of the deep Q network model and intelligent agent for cross-domain routing decision-making; (2) the stage of making routing decisions based on the deployed deep Q network model and intelligent agent.

[0068] First, the deep Q network model and agent preparation stage for cross-domain routing decision-making are introduced, which includes the following five steps.

[0069] Step 1: Model the satellite network environment and generate a state space, where the state space is a set used to describe all possible states of a system.

[0070] For newly arrived services, record the service request information: source node, destination node, duration, service demand width, and collect network information: link bandwidth information, direction perception information, network topology information, and generate the current state space together;

[0071] The state vector of Agent i is defined as Where D i，k It records the directional perception information of the current domain and the adjacent domain. It is calculated through the current control node prediction and the direction information Dir (A, D) of the target domain and the position information of the adjacent domain. The value close to the target domain is positive, and the value far away is negative. Indicates the link transmission distance between the current domain d and the adjacent domain d`. Indicates the link bandwidth between the current domain d and the adjacent domain d`.

[0072] Step 2: Design the action space. The action space is the set of all possible actions that the agent can perform in reinforcement learning. In this method, the agent can be the core control satellite.

[0073] Action space of agent i: Define the action space of agent i as Ai = Vi, where Vi is the set of neighboring agents of agent i. Therefore, for each agent, the size of the action space is the number of neighboring agents. Assume that packet k arrives at the head of the receiving queue at time t, agent i responds according to the current state S i，t and A i Select the forwarding action S i，t .

[0074] Step 3: Based on the state and action, design a reward mechanism to optimize the goal.

[0075] Reward function: In order to optimize the routing decision in the direction-aware assisted multi-agent hybrid routing algorithm, the reward of Agenti is defined as:

[0076]

[0077] Among them, α, β, γ are weight factors, v d is the target agent, Ψ is the positive incentive constant, and Φ is the negative penalty constant. BWsel is the bandwidth used by the current path, and BWmax is the maximum link bandwidth. D is introduced in the reward function i，k To drive the routing. Introduce direction perception information D into the reward function i，k The reward function optimizes the routing decision by combining direction-aware information, propagation delay, and bandwidth utilization in an integrated manner. Direction-aware information α·D i，k The positive reward of motivates the agent to choose a path closer to the goal; the propagation delay The negative return encourages the selection of low-latency paths; bandwidth utilization The positive return of improves the efficiency of bandwidth usage. The incentive Ψ and penalty Φ of specific conditions further optimize the decision of the intelligence in a given context. The overall design ensures that the intelligence makes efficient and expected routing choices in complex network environments.

[0078] Step 4: Build and train the dual deep Q network model and adjust the parameters of the model.

[0079] In order to more effectively capture the dynamic characteristics of the satellite network, a distributed training and distributed execution framework is adopted. The DQN network uses a neural network to approximate the state-action value function (Q-value function) and stores the agent's experience (including state, action, reward, next state, etc.) through experience replay (replaybuffer). The DQN model contains two neural networks with the same structure but different parameters: the target network (Qtar) and the evaluation network (Qeva). The target network is responsible for storing historical model parameters, which plays a role in stabilizing the training process and is updated regularly; the evaluation network evaluates the current state through the latest parameters and selects the optimal action for the newly arrived service request.

[0080] Step 5: Deploy the agent in the core satellite nodes and make routing decisions.

[0081] In the LEO satellite network, a hierarchical management architecture is adopted to divide the satellite into multiple domains and manage them through the core control satellite. The core control satellite monitors the network status and topology structure in the domain in real time, and exchanges information with neighboring domains to achieve global situation control. The inter-domain manager includes a state extraction module, a deep reinforcement learning (DRL) neural network, and a routing decision module. The state extraction module collects key information from the network, such as network topology, link bandwidth, and target domain direction perception information, and inputs it into the DRL neural network to generate the optimal cross-domain path. The domain manager is responsible for routing decisions within the domain. It obtains global data through centralized routing algorithms and controllers to calculate the optimal route within the domain to ensure the efficiency and accuracy of routing. To avoid local optimization problems, direction perception technology is introduced based on the location-assisted routing protocol (LAR). The core control satellite predicts the approximate direction of the target domain and integrates the direction information into the routing decision, effectively avoiding the long path and loop problems, simplifying the state space, and improving the efficiency of routing decisions.

[0082] On this basis, multi-agent hybrid routing based on direction perception is realized. Specifically, when a routing request arrives at the source node, the core control satellite uses the Dijkstra algorithm to process the intra-domain routing through the domain manager. If the target node is in another domain, the inter-domain manager collects network status information and combines the direction perception input deep reinforcement learning (DRL) model to perform cross-domain path calculation. The agent forwards the data hop by hop through distributed decision-making, and finally transmits it to the target domain gateway, and then the domain manager optimizes the intra-domain routing to complete the data transmission. This method combines distributed decision-making and centralized algorithms to achieve high efficiency and optimization of cross-domain and intra-domain routing.

[0083] The stage of making routing decisions based on the deployed deep Q network model and intelligent agent is introduced. Figure 2 The flowchart of the multi-agent hybrid routing method based on direction perception in another embodiment of the present invention is shown as follows. First, the satellite network is modeled and the status is collected. When the initiation and initial decision of the routing request are received, it is checked whether the target node is in the same domain. If so, the intra-domain routing decision is implemented based on the Dijkstra algorithm. If not, the cross-domain routing decision stage is entered. In the cross-domain routing decision stage, the inter-domain manager first collects status information, and then makes a cross-domain routing decision based on the direction perception information, forwarding the data packet hop by hop to the gateway node of the target domain, and the target domain manager executes the intra-domain routing decision. When everything is completed, the data is transmitted to the target node and the routing is completed.

[0084] The stage of making routing decisions based on the deployed deep Q network model and intelligent agent includes the following five steps:

[0085] Step 1: Satellite network modeling and status collection.

[0086] In the implementation process of the present invention, a LEO satellite network is first modeled, and a satellite network consisting of multiple satellite domains D = {1, 2, ..., 6} is considered. For each domain d, a satellite node set N is defined. d ={1, 2, ..., 16}, and establish a link set within the domain Therefore, the satellite links between domains Ensure that each satellite maintains contact with satellites in adjacent domains. In addition, a set of link distance parameters are obtained in domain d simulation And a set of cross-domain link distance parameters: To manage these satellites, they are divided into 6 satellite domains, each containing 16 satellites. Each satellite is connected to its 4 neighboring satellites in the network (two co-orbit neighbors and two adjacent orbit neighbors) through bidirectional links.

[0087] After modeling is completed, define the state space Its state vector includes the following information: direction perception information D i,k and link distance and link bandwidth

[0088] In a satellite network, the location and direction information of a node is three-dimensional. Considering the circumferential nature of the satellite network, the direction needs to be determined more accurately. In order to better reflect the direction perception information, we need to consider not only the angle, but also the direction (clockwise or counterclockwise). i,k It can be defined as:

[0089] D i，k =cos(θ i , k)·sgn((P k -P i )×(P D -P i ));

[0090] θ i , k is the direction vector from the current control node i to the adjacent domain node k and the direction vector v from the current control node i to the destination domain D → The angle between i and D. P i , P k and PD represent the position vectors of the control node i, the neighboring node k, and the destination domain D, respectively. The sign function sgn is used to determine whether the direction is clockwise or counterclockwise. i，k The value of reflects the relationship between the direction from control node i to adjacent node k and the direction to the destination domain D. i，kThe value of reflects the direction from the controller node i to the neighboring node k and the direction to the destination domain. Whether the direction is clockwise or counterclockwise, select D i，k Paths with larger absolute values ​​will lead to more effective approach to the destination domain D.

[0091] Step 2: Initiation of routing request and initial decision.

[0092] In the initiation and initial decision phase of routing requests, after receiving a new request, the source node will send it to the core control satellite of its domain. Figure 3 As shown in the figure, the core control satellite is a key node for routing management, and is deployed with a domain manager and an inter-domain manager, which are responsible for intra-domain and inter-domain routing processing respectively. Figure 3 The system block diagram of the multi-agent hybrid routing method based on direction perception in one embodiment of the present invention. The left side is the physical architecture of the satellite network, the lower left side is the ground network, the upper left side is the LEO satellite network environment, each dotted box is a satellite in a subdomain, and the core control satellite is located at the center of each subdomain. The figure illustrates bidirectional links, inter-domain links, intra-domain links and control links. The right side is the domain manager and inter-domain manager deployed on the core control satellite. Each domain manager contains an SDN controller and a centralized routing algorithm program for centralized method to make intra-domain routing decisions. Each inter-domain manager contains a state extractor for state information acquisition and a DRL neural network for cross-domain routing decisions.

[0093] In the specific implementation, the domain manager first collects the status information and service characteristics of the satellite network (such as link bandwidth, network load, etc.) to ensure the accuracy of the routing calculation. Then, it determines whether the target node is in the same domain. If so, a centralized algorithm (such as the Dijkstra algorithm) is used to calculate the optimal path in the domain to meet the quality of service (QoS) requirements and ensure the efficiency of routing. If the target node is located in another domain, the inter-domain manager takes over the task and starts the cross-domain routing mechanism. This initial judgment step is crucial. It quickly determines the routing strategy and ensures efficient and flexible data transmission. This example selects the case where the target node is not in the same domain as the source node for introduction.

[0094] Step 3: The inter-domain manager collects status information.

[0095] First, the inter-domain manager collects direction-aware information from neighboring inter-domain nodes through a flooding mechanism, including network status feedback such as link bandwidth, latency, and load. The state information after flooding will be used to determine the approximate direction of the domain where the target node is located. In order to reduce the consumption of network resources by frequent flooding, subsequent routing requests can be derived based on historical data obtained from previous flooding.

[0096] Assuming that the orbit of the target domain in the satellite network is known, the relative position change of the target domain can be predicted by combining the satellite trajectory parameters and historical link quality data. This derivation process can be completed by the time-weighted average method. Let the link delay in the flooding result be D i , bandwidth is B i , where i represents the i-th flooding, and the flooding result closest to the current time is given a higher weight.

[0097] The weighted average calculation formula for direction perception information is:

[0098]

[0099] Among them, ω i is the weight of the ith flood, which is usually set in a time-decreasing manner. The closer the flood result is to the current time, the higher the weight. The weighted average link delay calculated and bandwidth The relative direction of the domain where the current target node is located can be derived.

[0100] Step 4: The DRL agent makes cross-domain routing decisions.

[0101] A dual-depth Q network model was established and trained, with the decay value γ of future rewards set to 0.99, the learning rate α of the error set to 0.0001, and the random exploration rate ε set to 0.1.

[0102] Based on the collected direction perception information, the inter-domain manager inputs these network status data into the deep reinforcement learning (DRL) model. According to the real-time network status and direction information, the DRL model selects the optimal next-hop node through the decision-making strategy obtained through training. At each selection, the agent will give priority to those nodes with better network conditions and facing the target domain to ensure that the data packet moves gradually towards the target domain. When the data packet is forwarded hop by hop to the gateway node of the target domain, it proceeds to the next step.

[0103] The inter-domain manager combines the collected direction perception information with the trained agent decision strategy to select the optimal next-hop node. The agent uses a deep reinforcement learning (DRL) algorithm that not only relies on real-time network conditions, but also accumulates knowledge of historical link conditions through continuous learning, so that it can predict future link changes. This decision-making process gives priority to nodes with good network performance (such as high bandwidth and low latency) and facing the target domain, ensuring that the data packet gradually approaches the target domain in each jump. This distributed routing decision-making mechanism makes cross-domain routing more flexible by utilizing inter-domain agents, and can adaptively respond to different network conditions and topology changes.

[0104] Step 5: The target domain manager performs intra-domain routing decisions.

[0105] After the data packet is forwarded hop by hop and finally reaches the gateway node of the target domain, the task of cross-domain routing is completed, and the intra-domain routing decision phase is entered. The core control satellite of the target domain takes over the data packet, and the domain manager deployed inside it is responsible for executing the intra-domain routing decision. The specific implementation details are as follows: First, the domain manager will immediately collect the network status information in the current target domain, including link bandwidth, latency, load status, etc., to ensure that the network topology and link quality are fully understood. Next, the domain manager uses a centralized algorithm (such as the Dijkstra algorithm) to calculate the intra-domain path based on the location and network status of the target node. The Dijkstra algorithm can quickly find the optimal path from the gateway node to the target node based on the weight of each link (such as bandwidth, latency, etc.), ensuring that the path meets the Quality of Service (QoS) requirements.

[0106] Once the data packet arrives at the gateway node of the target domain, the system enters the intra-domain routing phase. In this phase, the routing mechanism switches from distributed to centralized. The core control satellite of the target domain immediately obtains and updates the network status information in the target domain, such as link bandwidth, latency, load, etc., through the domain manager deployed inside it. By fully grasping this real-time information, the domain manager can quickly make intra-domain routing decisions. Unlike inter-domain routing, the intra-domain uses traditional centralized algorithms (such as the Dijkstra algorithm). The Dijkstra algorithm can quickly find the optimal path from the gateway node to the target node based on the weight of each link (such as bandwidth, latency, etc.), ensuring that the path meets the quality of service (QoS) requirements.

[0107] During the calculation process, the domain manager will comprehensively consider the real-time status of the link and give priority to links with sufficient bandwidth, low latency and balanced load to ensure that the data packet can be efficiently transmitted to the target node. Specifically, the Dijkstra algorithm calculates the shortest path P(s, t) from the gateway node s to the target node t based on the link weight ω(e) in the network graph to ensure that the selected path P meets the optimality:

[0108] P(s, t) = min∑ e∈P ω(e);

[0109] Through this path calculation, the domain manager finally determines an optimal transmission path and transmits the data packet hop by hop along the path to the target node. Once the data packet successfully reaches the target node, the entire routing process ends and the intra-domain routing decision is successfully completed.

[0110] As mentioned above, the present invention introduces direction perception information based on the location-assisted routing protocol (LAR) in the training process of the intelligent agent. By incorporating this information into the reward function of the intelligent agent, the local optimal solution caused by local observation and the non-stationary problem caused by frequent changes in network topology can be effectively avoided, thereby improving the stability and reliability of the overall routing. Through this method, the present invention can improve the adaptability and global optimization ability of routing decisions in a dynamic environment while ensuring low communication overhead, and is suitable for complex satellite network communication environments.

[0111] The multi-agent hybrid routing method proposed in this invention can combine the intelligent routing mechanisms of centralized and distributed mechanisms to effectively cope with the complex routing requirements in large-scale satellite networks. By intelligently combining these two routing mechanisms, the flexibility of the distributed mechanism and the global optimization capability of the centralized mechanism are fully utilized, so that the entire network can efficiently and dynamically handle cross-domain and intra-domain routing tasks.

[0112] On the one hand, in the cross-domain routing process, the system implements distributed intelligent routing decisions through the deep reinforcement learning (DRL) model. Each inter-domain manager, as an independent intelligent agent, selects the optimal next-hop node through the DRL model based on the collected network status data, direction perception information and other inputs. This distributed decision-making method can respond to changes in network status in real time, including link bandwidth, latency, load, etc., to ensure that data packets gradually move toward the target domain, avoiding single point failure problems that may exist in traditional centralized routing. At the same time, the DRL model can effectively accumulate historical data through continuous training and learning, thereby predicting the status of future links, further improving the accuracy and reliability of decision-making.

[0113] On the other hand, when the data packet arrives at the gateway node of the target domain, the system switches to a centralized routing mechanism. The core control satellite of the target domain comprehensively collects network status information in the current domain through the domain manager, including link bandwidth, latency, load, etc., and uses classic algorithms such as Dijkstra to perform global optimization calculations of the paths within the domain. Compared with distributed decision-making, centralized algorithms can quickly find the optimal path from the gateway node to the target node under a known network topology, ensuring that the path meets the quality of service (QoS) requirements, such as low latency, high bandwidth, and load balancing. This centralized routing decision effectively utilizes the global view of the network and can ensure the selection of the optimal path within a small-scale topology.

[0114] Finally, the effectiveness of the method is verified through experiments and compared with the existing technology.

[0115] The experimental conditions are as follows: In an environment where the CPU model is Intel(R) Xeon(R) Gold 5218R CPU@2.10GHz, we used the MATLAB AERO toolbox in Matlab2022 to simulate a Walker constellation and used the reinforcement learning toolbox to implement agent training.

[0116] In this paper, we designed and compared three routing algorithms covering distributed intelligent routing and centralized routing paradigms. These include the centralized shortest path routing algorithm (SR-C), the distributed shortest path routing algorithm (SR-D), and the distributed routing algorithm based on reinforcement learning, the Q-routing algorithm. The experimental results are as follows:

[0117] (1) Figure 4 As shown, Figure 4 This is a convergence diagram of the reward function of the DAMHR algorithm in one embodiment of the present invention. The real part is the result of taking the average value of the shaded part every ten times. It can be seen from the figure that the DAMHR algorithm converges quickly and has small fluctuations. After the algorithm reaches convergence, the algorithm's latency and bandwidth utilization are evaluated.

[0118] (2) Figure 5 The end-to-end delay performance of the four algorithms under different loads is shown. Figure 5 The figure is a comparison diagram of the end-to-end delay index between the method of the present invention and the traditional method. As the load increases, the delay of the DAMHR algorithm is significantly lower than that of other algorithms, and it shows high robustness, and its delay drops rapidly and remains stable. In contrast, the delay of the SR-C algorithm rises sharply when the load exceeds 300Mbps, while the delay of the SR-D algorithm increases slowly but limitedly, especially under high load conditions. In contrast, the Q-routing algorithm shows a continuously high delay in the entire load range, and is particularly prominent under high load.

[0119] (3) Figure 6 The figure shows how bandwidth utilization changes with load. The DAMHR algorithm significantly improves bandwidth utilization and maintains high stability under various load conditions. Figure 6 The figure is a comparison diagram of the bandwidth utilization index between the method of the present invention and the traditional method. In comparison, the bandwidth utilization of the SR-C algorithm decreases significantly under high load conditions, while the SR-D algorithm shows a slight decrease in bandwidth utilization after the load gradually increases. The bandwidth utilization of the Q-routing algorithm is always at a low level. Therefore, the DAMHR algorithm shows superior performance and higher robustness in terms of both latency and resource utilization.

[0120] Corresponding to the above method, the present invention also provides a multi-agent hybrid routing system based on direction perception assistance, the system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the above method. The above system is carried on the computer device of the core control satellite.

[0121] In summary, the key points and advantages of the multi-agent hybrid routing method based on direction perception proposed in the present invention are:

[0122] (1) Through the proposed domain-based and hierarchical routing calculation strategy, the routing task is divided into two parts: intra-domain and inter-domain, and different algorithms are used to process them respectively. Among them, the inter-domain routing uses an intelligent agent trained based on deep reinforcement learning (DRL) to make distributed routing decisions, and the intra-domain routing uses centralized routing decisions to improve the calculation efficiency and the accuracy of path selection.

[0123] (2) During the training process of the agent, by introducing directional perception information based on spatial location, the adaptability and efficiency of routing decisions in dynamic network environments can be improved. By incorporating this directional perception information into the agent's reward function, the local optimal solution caused by local observations and the non-stationary problem caused by frequent changes in network topology can be effectively avoided, thereby improving the stability and reliability of the overall routing.

[0124] (3) By adopting a new routing mechanism that combines centralized and distributed routing, the intra-domain routing and inter-domain routing tasks are cleverly separated, and optimized routing strategies are designed for both. Within the domain, a centralized routing algorithm is used to improve computational efficiency and ensure that path selection is completed quickly and accurately within a small range; while between domains, a distributed routing algorithm based on deep reinforcement learning (DRL) is used to enable the intelligent agent to dynamically adapt to the frequent changes in the satellite network topology. This combined strategy ensures the effective integration of global views and local information, avoids the limitations of a single algorithm, and not only improves the overall adaptability and flexibility of the network, but also reduces communication overhead.

[0125] (4) By introducing a multi-agent routing algorithm assisted by direction perception, the direction perception information of the spatial position is incorporated into the training process of the agent, which enables the agent to make more adaptive routing decisions in a complex and dynamic satellite network environment. This direction perception information provides a more refined perception of the topological structure, enabling the agent to identify potential directional transmission needs in the network and effectively avoid the generation of local optimal solutions. In addition, the introduction of this information enhances the system's robustness to frequent topological changes, improves the stability of the algorithm and the overall efficiency of network transmission, especially in a dynamic environment, it can better adapt to changes and achieve better path planning.

[0126] (5) Through the global perspective of the multi-agent system, the load balancing capability of the satellite network is significantly improved, and the problem of uneven resource allocation caused by insufficient local information in traditional routing algorithms is solved. The agent can monitor the traffic distribution, path load and bandwidth usage of the entire network in real time, make dynamic decisions based on a global perspective, and effectively avoid the overload problem of hot spots. When the agent finds that the traffic on some links is too large, it will intelligently disperse the traffic to paths with lighter loads, thereby reducing the probability of communication bottlenecks. This global optimization strategy not only improves the overall throughput of the network, but also improves the stability and communication efficiency of the system in complex dynamic environments. At the same time, the agent has adaptive capabilities and can optimize resource allocation in real time according to changes in network load, ensuring low latency and high bandwidth utilization under high load or emergency conditions, greatly enhancing the robustness and adaptability of the network.

[0127] (6) As described above, the method proposed in the present invention can improve the adaptability and global optimization ability of routing decisions in a dynamic environment while ensuring low communication overhead, and is suitable for complex satellite network communication environments.

[0128] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0129] See also Figure 7 , Figure 7The structure of a computer device in a multi-agent hybrid routing system in one embodiment of the present invention is shown in FIG. The computer device 00 includes a processor 01, a memory 02, and a computer program stored in the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, the human factor data server access control method provided in the above method embodiment is implemented.

[0130] Among them, the processor 01 is connected to the memory 02, such as through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The bus 03 may include a path to transmit information between the above components. The bus 03 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 02 is used to store a computer program corresponding to the human factor data server access control method of the above embodiment of the present application, and the computer program is controlled and executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to implement the content shown in the above method embodiment.

[0131] Corresponding to the above method, the present invention also provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0132] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0133] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0134] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0135] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-agent hybrid routing method based on direction perception assistance, characterized in that: The satellite network is pre-divided into multiple sub-domains, each sub-domain includes a core control satellite, and the satellite is used as a node of the satellite network. A domain manager and an inter-domain manager are pre-deployed on the core control satellite, respectively. The method includes: In a pre-built satellite network environment model, the domain manager is used to collect status information within the domain, and the inter-domain manager is used to collect direction perception information and inter-domain status information from adjacent inter-domain nodes through a flooding mechanism; wherein the status information includes multiple types of link bandwidth, link delay, link load and network topology information; The core control satellite receives data packets to be forwarded and corresponding routing requests; the data packets come from satellites in the subdomain or core control satellites in other subdomains; When the core control satellite parses the routing request and determines that the data packet is in the inter-domain routing stage, it uses the pre-trained direction-aware assisted deep reinforcement learning neural network contained in the inter-domain manager to make cross-domain routing decisions; When the core control satellite parses the routing request and determines that the data packet arrives at the target domain, the domain manager deployed on the core control satellite is used to make intra-domain routing decisions according to the centralized routing method; The reward function of the direction perception-assisted deep reinforcement learning neural network integrates direction perception information and state information in an integrated manner to optimize routing decisions.

2. The method according to claim 1, characterized in that The method further comprises: Before making a routing decision, a satellite network environment model is pre-built for the satellite network in the low earth orbit, and a state space is generated after the satellite network environment model is built, wherein the state space includes the state information within the domain, the state information between the domains, and the direction perception information; During the routing process, for a newly arrived routing request, service request information is recorded, where the service request information includes multiple information such as a source node, a destination node, a duration, and a service requirement width.

3. The method according to claim 1, characterized in that The direction perception information is derived based on the location-assisted routing protocol through historical trajectory parameters and historical link quality data of the satellite; wherein the historical link quality data includes link delay and link bandwidth.

4. The method according to claim 1, characterized in that: The direction perception information is fused with the angle and direction information and is defined in the form of a formula as follows: D i,k ncos(θ i ,k)·sgn((P k -P i )×(P D -P i )) Among them, D i,k is the direction perception information, θ i ,k is the direction vector from the current control node i to the adjacent domain node k and the direction vector from the current control node i to the destination domain D The angle between i , P k and P D They represent the position vectors of the control node i, the adjacent node k and the destination domain D respectively. The sign function sgn is used to determine whether the direction is clockwise or counterclockwise.

5. The method according to claim 1, characterized in that The type of the deep reinforcement learning neural network is a deep Q learning network. The model parameters that need to be set in the initial stage of training the deep Q learning network include a decay value for future rewards, a learning rate for errors, and a random exploration rate. The state vector of the deep Q learning network includes direction perception information, link distance, and link bandwidth. During the training process of the deep Q learning network, state information and direction perception information are used as input, and the next hop node of the routing strategy is used as output.

6. The method according to claim 5, characterized in that The step of using the pre-trained direction-awareness-assisted deep reinforcement learning neural network contained in the inter-domain manager to make cross-domain routing decisions includes: The latest collected direction perception information and state information are input into the direction perception-assisted deep reinforcement learning neural network to obtain the next hop node of the output cross-domain routing strategy.

7. The method according to claim 5, characterized in that The reward function of the deep reinforcement learning neural network assisted by direction perception is defined in formula form as follows: Among them, α, = and γ are weight factors, v d is the target agent, Ψ is the positive incentive constant, Φ is the negative penalty constant, D i,k is the direction perception information, is the link delay obtained by dividing the link distance by the speed of light, a i represents the current agent, BWsel is the bandwidth used by the current path, and BWmax is the maximum link bandwidth.

8. A multi-agent hybrid routing system based on direction perception assistance, comprising a processor, a memory and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images