A structured action value network for variable neighbor set, routing decision method and device
Patent Information
- Application Number
- CN202611390477.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-09
- Publication Date
- 2026-10-30
AI Technical Summary
虽然这类方法能够表征图结构信息，但在拓扑快速变化的动态网络中，频繁获取大范围拓扑状态会带来较高的通信开销和状态同步延迟；同时，多轮邻接聚合增加了推理时延和计算资源消耗，难以在卫星载荷、车载终端或传感器节点等资源受限的设备上实时运行
[0034]第一，本发明对于不同有效候选特征采用共享的编码器参数，使评价规则绑定有效候选特征自身，即使候选邻居顺序和数量产生变化，也只改变输出的顺序和数量，不会改变评价结果，有助于减少位置伪相关，能够适应候选邻居数量和顺序的变化；
Smart Images
Figure CN123020443A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer network routing control technology, and in particular to a structured action value network for variable neighbor sets, a routing decision method and apparatus. Background Technology
[0002] In communication networks (such as low-Earth orbit satellite constellation networks, mobile ad hoc networks, vehicle-to-everything (V2X) networks, wireless sensor networks, and edge computing networks), routing decisions are crucial for ensuring efficient data transmission. In these networks, due to the high-speed movement of nodes, changes in link status with the environment, and nodes joining or leaving the network at any time, the number and order of candidate next-hop neighbors available to each node at the moment of decision-making are often dynamically changing. Taking low-Earth orbit satellite networks as an example, satellites move at high speeds relative to the ground, and inter-satellite links are established and disconnected periodically with orbital phase changes. A satellite may establish links with multiple neighboring satellites simultaneously in the current time slot, while its visible neighbor set may increase or decrease in members or change in order in the next time slot. Similarly, in mobile ad hoc networks and V2X networks, a node's communication neighbors are determined by its relative location and signal coverage, and the neighbor set changes rapidly as the node moves.
[0003] Existing routing decision-making methods based on deep reinforcement learning typically concatenate the current node state, destination node information, and the states of each neighbor into a fixed-length feature vector, which is then input into a Deep Q-Network (DQN). The fixed-dimensional output layer then provides the action value of each candidate action. To adapt the network to a variable number of candidate neighbors, these methods usually pre-define a maximum number of neighbors and fill each neighbor into a predetermined slot with a fixed number. If the number of neighbors is insufficient, zero values are used to fill the slots; if the number exceeds the limit, the network is truncated, retrained, or its structure is modified.
[0004] However, the aforementioned existing technologies have significant drawbacks. First, the fixed-slot design introduces positional spurious correlations—the network tends to use the position of a specific slot in the input vector, rather than the actual state of its corresponding neighbor, as the basis for action evaluation. When the neighbor enumeration order changes, the same neighbor may fall into different slots, causing changes in the output action value and reducing the stability and reliability of the decision. Second, it lacks adaptability to changes in the number of candidates—filling in a large number of invalid terms increases computational redundancy, and when the actual number of candidates exceeds the preset maximum value, the network cannot process it directly and must be redesigned and retrained. Third, the number of model parameters is tied to the preset maximum number of neighbors or the total number of nodes in the network. When the network scales up or is deployed to networks of different sizes, the network structure needs to be adjusted and retrained, making it difficult to achieve cross-node, cross-topology, and cross-scale migration deployments.
[0005] In addition, another approach leverages graph neural networks for message passing across the entire graph or multi-hop neighborhood to gain global topology awareness and output routing actions. While these methods can characterize graph structure information, in dynamic networks with rapidly changing topologies, frequent acquisition of large-scale topology states leads to high communication overhead and state synchronization delays. Furthermore, multi-round adjacency aggregation increases inference latency and computational resource consumption, making it difficult to run in real-time on resource-constrained devices such as satellite payloads, vehicle terminals, or sensor nodes. Graph neural networks also face issues of over-smoothing and over-compression; deep message passing can cause node features to become uniform and lose their distinctiveness, hindering effective learning in large-scale or deep networks.
[0006] Therefore, there is an urgent need for an action value calculation scheme that can directly handle a variable number of candidate neighbor sets with arbitrary order, whose model parameters do not increase with the total number of network nodes, and which can complete routing decisions without relying on full graph message passing. Summary of the Invention
[0007] In view of the above problems, embodiments of this application provide a structured action value network, routing decision method, apparatus, electronic device, and readable storage medium for variable neighbor sets, so as to overcome the above problems or at least partially solve the above problems.
[0008] In a first aspect, embodiments of this application provide a structured action value network and a routing decision method for variable neighbor sets, the method comprising:
[0009] Based on the routing state, basic features are generated for the current node, and candidate features are constructed for each candidate neighbor of the current node. Based on the information in the candidate features, the candidate neighbors are divided into valid candidate neighbors and invalid candidate neighbors, thereby generating a set of validity masks describing the validity of the candidate neighbors.
[0010] The basic features and the candidate features of effective candidate neighbors are encoded by the basic state encoder and the neighbor encoder respectively to obtain the basic embedding and the effective candidate embedding. The basic embedding and the effective candidate embedding are combined into a combined feature. The action evaluation network is used to calculate the advantage value of the combined feature, and the value head network is used to calculate the state value of the basic embedding.
[0011] The number of valid candidate neighbors is determined based on the set of validity masks. The mean advantage of the valid candidate neighbors is calculated by combining the advantage value. The action value of the valid candidate neighbors is calculated using the state value, advantage value, and mean advantage value. The valid candidate neighbor with the highest action value is used as the next hop for the routing decision.
[0012] Optionally, the step of generating basic features for the current node based on routing state, constructing candidate features for each candidate neighbor of the current node, and classifying candidate neighbors into valid and invalid candidate neighbors based on information in the candidate features, thereby generating a set of validity masks describing the validity of candidate neighbors, specifically including:
[0013] Based on the current routing state, generate the basic characteristics of the current node;
[0014] List the candidate neighbors of the current node, construct candidate features for each candidate neighbor of the current node, and form a candidate feature set from multiple candidate features;
[0015] Based on the candidate features of each candidate neighbor, valid candidate neighbors and invalid candidate neighbors are determined, and a validity mask is assigned to each valid candidate neighbor and invalid candidate neighbor to generate a set of validity masks.
[0016] Optionally, the process of encoding the basic features and the candidate features of effective candidate neighbors using a basic state encoder and a neighbor encoder respectively to obtain basic embeddings and effective candidate embeddings, combining the basic embeddings and effective candidate embeddings into a combined feature, calculating the combined feature as an advantage value using an action evaluation network, and calculating the basic embedding as a state value using a value head network, specifically includes:
[0017] The basic features are input into the basic state encoder to obtain the basic embedding;
[0018] Input the candidate features of the effective candidate neighbors into the neighbor encoder to obtain multiple effective candidate embeddings;
[0019] Based on the combination operator, the basic embedding and the candidate embeddings of each effective candidate neighbor are combined into multiple combined features;
[0020] The combined features of effective candidate neighbors are input into the action evaluation network to obtain the advantage value of each effective candidate neighbor.
[0021] The basic embedded input value head network is used to obtain the state value of the current node.
[0022] Optionally, the step of determining the number of valid candidate neighbors based on the validity mask set, calculating the mean advantage of valid candidate neighbors in combination with the advantage value, and calculating the action value of valid candidate neighbors using the state value, advantage value, and mean advantage; and selecting the valid candidate neighbor with the highest action value as the next hop for the routing decision, specifically includes:
[0023] The number of valid candidate neighbors is obtained by adding the validity masks in the validity mask set.
[0024] Add up the dominance values of all valid candidate neighbors and then divide by the number of valid candidate neighbors. Take the average of the dominance values to obtain the mean dominance value of the valid candidate neighbors.
[0025] Based on the current node's state value, the advantage value of each valid candidate neighbor, and the average advantage value, calculate the action value of each valid candidate neighbor.
[0026] Choose the nearest valid candidate neighbor with the highest action value as the next hop.
[0027] Secondly, embodiments of this application provide a structured action value network and a routing decision device for a variable neighbor set, used to implement the method described in the first aspect, the device comprising:
[0028] The feature construction module is used to generate basic features for the current node based on the routing state, construct candidate features for each candidate neighbor of the current node, and divide the candidate neighbors into valid candidate neighbors and invalid candidate neighbors based on the information in the candidate features, thereby generating a set of validity masks describing the validity of the candidate neighbors.
[0029] The encoding and computation module is used to encode the basic features and the candidate features of the effective candidate neighbors using the basic state encoder and the neighbor encoder respectively, to obtain the basic embedding and the effective candidate embedding respectively. The basic embedding and the effective candidate embedding are combined into a combined feature. The action evaluation network is used to calculate the advantage value of the combined feature, and the value head network is used to calculate the state value of the basic embedding.
[0030] The selection module is used to determine the number of valid candidate neighbors based on the set of validity masks, calculate the average advantage of valid candidate neighbors by combining the advantage value, and calculate the action value of valid candidate neighbors using the state value, advantage value and average advantage value; and select the valid candidate neighbor with the largest action value as the next hop for the routing decision.
[0031] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the structured action value network and routing decision method for variable neighbor sets as described above.
[0032] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the structured action value network and routing decision method for a variable neighbor set as described above.
[0033] The specific beneficial effects are as follows:
[0034] First, the present invention uses shared encoder parameters for different effective candidate features, so that the evaluation rules are bound to the effective candidate features themselves. Even if the order and number of candidate neighbors change, only the order and number of outputs will change, and the evaluation results will not change. This helps to reduce positional spurious correlations and can adapt to changes in the number and order of candidate neighbors.
[0035] Second, the validity mask set can ensure that non-existent or invalid candidate neighbors do not participate in the calculation and selection, reducing invalid inputs and computational load;
[0036] Third, each action value corresponds clearly to a candidate feature, i.e. a candidate neighbor. The advantage value, mask, and features of the candidate neighbor can be analyzed separately, which facilitates fault location and strategy auditing.
[0037] In summary, this invention addresses the problems of changes in evaluation results due to neighbor node rearrangement and increased invalid computation due to padding caused by changes in the number of neighbor nodes in a network. It decomposes the current routing state into basic features and multiple candidate features, and constructs a set of validity masks. Based on the basic features, candidate features, and validity masks, the action value of each neighbor node is calculated, and the next hop of the route is determined according to the action value. This application can adapt to changes in the number and order of neighbors, helps reduce spurious location correlations, reduces invalid inputs and computations, lowers the dependency on full-graph message passing, facilitates engineering implementation and interpretation, and provides a reliable solution for routing decisions in communication networks. Attached Figure Description
[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating a structured action value network and routing decision method for a variable neighbor set provided in an embodiment of this application.
[0040] Figure 2 This is a schematic diagram of the action value network structure provided in the embodiments of this application;
[0041] Figure 3 This is a diagram showing the results of determining the effectiveness of the local topology and candidate neighbor mask according to an embodiment of this application;
[0042] Figure 4 This is a statistical chart of the action value of effective candidate neighbors provided in an embodiment of this application. Detailed Implementation
[0043] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.
[0044] Reference Figure 1 , Figure 1 This application provides a flowchart illustrating a structured action value network and routing decision method for a variable neighbor set, which may include:
[0045] Step 101 generates basic features for the current node based on the routing state, constructs candidate features for each candidate neighbor of the current node, and divides the candidate neighbors into valid candidate neighbors and invalid candidate neighbors based on the information in the candidate features, thereby generating a set of validity masks describing the validity of the candidate neighbors.
[0046] The current node is the node where the data packet is currently located in the communication network. In some embodiments, the communication network can be a low-Earth orbit satellite network, an inter-satellite link network, a mobile ad hoc network, a vehicle-to-everything (V2X) network, a wireless sensor network, an edge computing network, or a general communication network. Nodes that have established a direct communication link with the current node (without passing through other nodes) and are allowed as potential next hops according to communication protocols, interface capabilities, service permissions, or forwarding policies can be called the current node's candidate neighbors. The current node's candidate neighbors form a candidate neighbor set. The number of candidate neighbors in this set, as well as their order in the engineering interface, may change as the current node moves. The current node can send data packets directly to candidate neighbors without passing through other nodes; the corresponding candidate neighbor set may also be different for different data packets.
[0047] Optionally, step 101 may include the following sub-steps:
[0048] Sub-step 1011: Based on the routing state at the current moment, generate the basic features of the current node.
[0049] At some point The current node possesses several routing states, such as its identifier, location, destination node's location (the node the data ultimately reaches in the communication network), current time, service type, remaining packet timeout, current node's buffer queue length, and computing resources. These routing states are specific to the current node and apply to all its candidate neighbors. Therefore, the time can be... The routing state constitutes the basic characteristics of the current node. Its dimension number is .
[0050] Sub-step 1012: List the candidate neighbors of the current node, construct candidate features for each candidate neighbor of the current node, and form a candidate feature set from multiple candidate features.
[0051] List the candidate neighbors of the current node in the communication network. The current node at time... A node can have multiple candidate neighbors. All candidate neighbors form the current node's candidate neighbor set, and can be arranged into a candidate neighbor list according to the order given by the engineering interface. The link state, neighbor load, and subsequent forwarding conditions between the current node and a candidate neighbor can be described by the candidate characteristics of that candidate neighbor. Candidate characteristics can include the following fields:
[0052] Relative position: The difference between the current node's position and the position of the candidate neighbor. The position of the candidate neighbor can be obtained from beacons, global navigation satellite system (GNSS) announcements, or satellite ephemeris announcements.
[0053] Link distance: Calculated based on the coordinates of the current node and the candidate neighbor, or obtained by measuring the round-trip time of communication;
[0054] Propagation delay: Calculated by dividing the link distance by the propagation speed, or measured by timestamped probe messages;
[0055] Available bandwidth: obtained by subtracting the occupied bandwidth from the interface capacity;
[0056] Bit error rate: measured and reported by the physical layer or link layer;
[0057] Signal-to-noise ratio: Measured and reported by the physical layer or link layer;
[0058] Number of forwardable next hops: The number of next-hop nodes that the candidate neighbor can currently connect to and is allowed to forward to;
[0059] Remaining cache resources: obtained based on the state of the candidate neighbor;
[0060] Estimated distance to the target node: calculated from the positions of the current node and the target node, ephemeris, or local topology / routing table;
[0061] Link duration: predicted from the relative motion trajectory of the current node and the candidate neighbor or from the historical link status;
[0062] Reachability routing table: Obtained based on the routing table, communication link status, or directional constraints;
[0063] Historical forwarding statistics: obtained based on the candidate neighbor's local records of sending success rate, retransmission count, average latency, and packet loss rate.
[0064] It should be noted that the order of fields in the candidate features may not be the same for different candidate neighbors.
[0065] Each candidate neighbor can form a candidate feature, and multiple candidate features form a candidate feature set.
[0066] Sub-step 1013: Based on the candidate features of each candidate neighbor, determine the effective candidate neighbors and the invalid candidate neighbors, assign validity masks to the effective candidate neighbors and the invalid candidate neighbors, and generate a set of validity masks.
[0067] Each candidate neighbor's candidate features describe the communication state of the communication link from the current node to that candidate neighbor. Therefore, the validity of a candidate neighbor can be determined based on its candidate features. For example, if the propagation delay or bit error rate of the communication link from the current node to a candidate neighbor is large, it indicates that the state of the communication link is poor, and the candidate neighbor is invalid and can be recorded as an invalid candidate neighbor. Conversely, if the propagation delay or bit error rate is large, the candidate neighbor is valid and can be recorded as a valid candidate neighbor.
[0068] In some embodiments, when for such Figure 2 When training or performing batch inference on the action value network shown, the candidate features of the current node's valid candidate neighbors can be organized into a shape of... The tensor, in which It refers to the batch size, i.e., the number of data packets. This refers to the number of candidate neighbors that the current node must have, as required by the actual engineering interface. This refers to the dimension of the candidate features. For example, the current node needs to... A data packet is sent out; in actual engineering interfaces, the current node must have [a specific function / requirement]. There are n candidate neighbors, and the dimension of the candidate features for each valid candidate neighbor of the current node is n. Due to tensors The current node must have There are several candidate neighbors, but the number of valid candidate neighbors for the current node may be insufficient. Therefore, empty spaces in the tensor can be filled with 0s, and non-existent candidate neighbors are set as invalid candidate neighbors (correspondingly, the corresponding validity mask in the validity mask set is also set to 1), thus forming a complete... Tensor. Tensor flattening is The input format is then used by subsequent neighbor encoders, encoded, and then restored to its original form. Form, in which This is the number of dimensions of the valid candidate embeddings output by the neighbor encoder in the following text. Therefore, invalid candidate neighbors of the current node include the two cases mentioned above.
[0069] Based on the above, the current node at time... If a candidate has multiple valid candidate neighbors and multiple invalid candidate neighbors, the validity mask for valid candidate neighbors is set to 1, and the validity mask for invalid candidate neighbors is set to 0. All validity masks are arranged in the corresponding order in the candidate neighbor list to form a validity mask set. .
[0070] Step 102: Encode the basic features and the candidate features of the effective candidate neighbors using the basic state encoder and the neighbor encoder respectively to obtain the basic embedding and the effective candidate embedding. Combine the basic embedding and the effective candidate embedding into a combined feature. Calculate the advantage value of the combined feature using the action evaluation network. Calculate the state value of the basic embedding using the value head network.
[0071] To determine the next hop of the current node, i.e., which valid candidate neighbor to send the data packet to, the action value of each candidate neighbor needs to be calculated using the action value network in this embodiment. For example... Figure 2 As shown, the input to the action value network is the basic features obtained in step 101, the candidate features of the effective candidate neighbors (denoted as effective candidate features), and the set of effectiveness masks.
[0072] The action value network provided in this application embodiment can be trained using reinforcement learning based on temporal difference. Each time the current node makes a decision, it stores the state of this decision and the state of the next time step into an experience replay buffer. When the current node is idle, a small batch of samples is randomly selected from the experience replay buffer for offline learning. Experience replay can be uniform replay or priority experience replay.
[0073] The action value network in this application embodiment can use Huber loss or mean squared error as a loss function to periodically or softly update the target network.
[0074] Optionally, step 102 may include the following sub-steps:
[0075] Sub-step 1021: Input the basic features into the basic state encoder to obtain the basic embedding.
[0076] like Figure 2 As shown, using the basic state encoder Basic features Encode to obtain the basic embedding In some embodiments, the basic state encoder It is A two-layer fully connected network can transmit basic features It is converted into a fixed-dimensional state representation. In some embodiments, the basic state encoder... First of all Perform field preprocessing. Field preprocessing is... Operations that unify the representation of different fields, such as basic features. Continuous fields (such as the remaining time limit for a group, which is a numerical value) can be normalized to the range [0,1]. For Boolean fields (such as fields representing "yes" or "no"), 1 can represent "yes" and 0 can represent "no". For categorical fields (such as business type), one-hot encoding or embedded encoding can be used (e.g., using [0,0,1] and [0,1,0] to represent different business types). After completing the field preprocessing, the preprocessed basic features are obtained. Basic state encoder right The basic embedding is calculated using the following formula. :
[0077] (Equation 1)
[0078] (Equation 2)
[0079] In Equation 1, It is the first intermediate vector. It is the Gaussian error linear unit activation function. It is the first weight parameter. It is the first bias vector; in Equation 2, The representation layer normalization function can calculate the mean and variance of its input along the feature dimension and normalize it, and then transform it through learnable scaling and translation parameters. It is the second weighting parameter. It is the second bias vector. Basic embedding. It is a dense numerical representation of common routing status such as current node location and resources, target node location, service type, remaining time limit of the packet, and current time, and can be used as input to the value head network and action evaluation network.
[0080] Basic state encoder parameters It can include Equation 1 and Equation 2 , , , ,as well as The scaling and translation parameters used by the function.
[0081] The above description of the dimension and number of layers of the basic state encoder is only an example, and the embodiments of this application do not make specific limitations.
[0082] Sub-step 1022: Input the candidate features of the effective candidate neighbors into the neighbor encoder to obtain multiple effective candidate embeddings.
[0083] In the set of validity masks In the middle, validity mask ( )express The index is The candidate neighbors corresponding to the validity mask are valid candidate neighbors. Therefore, based on the set of validity masks, the current node at time [time value missing] can be selected. All valid candidate neighbors in The indexes in the database are used to filter out and form a valid candidate set. As shown below:
[0084] (Equation 3)
[0085] Candidate features of all valid candidate neighbors (referred to as effective candidate features) form a set of effective candidate features. .
[0086] like Figure 2 As shown, using the neighbor encoder For each valid candidate feature ( Encode the valid candidate features to obtain the valid candidate embeddings for each valid candidate feature. A neighbor encoder can consist of one or more fully connected layers. In some embodiments, the neighbor encoder... It can be adopted It is a two-layer fully connected network, and the same network parameters are used for each valid candidate neighbor.
[0087] For each valid candidate feature Neighbor encoder First, field preprocessing is performed to obtain the preprocessed valid candidate features. The field preprocessing of the neighbor encoder adopts a method similar to that of sub-step 1021, but due to the effective candidate features... Included fields and basic features Different, therefore effective candidate features The field preprocessing performed uses a normalization method or encoding scheme corresponding to the field itself. After field preprocessing is completed, the neighbor encoder... Then, for effective candidate features Perform the following calculations:
[0088] (Equation 4)
[0089] In Equation 4, It is the second intermediate vector. It is the third weighting parameter. It is the third bias vector. Then, according to... Calculate the index as Valid candidate embeddings of valid candidate neighbors As shown below:
[0090] (Equation 5)
[0091] In Equation 5, It is the fourth weighting parameter. It is the fourth paranoia vector.
[0092] Effective candidate embeddings are valid candidate features Input into neighbor encoder The resulting fixed-dimensional dense vector comprehensively represents the link quality, neighbor load and remaining resources, forwarding conditions to the target node, and historical reliability of the effective candidate neighbors. Each dimension is a latent feature formed through training and is generally not assigned a fixed physical meaning on a dimension-by-dimensional basis. Each effective candidate neighbor corresponds to an effective candidate embedding. When the number of effective candidate neighbors changes, only the number of effective candidate embeddings changes, without changing the vector dimension or the number of encoder parameters.
[0093] Neighbor encoder parameters It can include Equations 4 and 5 , , , ,as well as The scaling and translation parameters used by the function.
[0094] The above description of the number of layers and dimensions of the neighbor encoder is only an example, and the embodiments of this application do not limit it.
[0095] Sub-step 1023: Based on the combination operator, the basic embedding and the candidate embeddings of each effective candidate neighbor are combined into multiple combined features.
[0096] like Figure 2 As shown, the basic embedding With each valid candidate embedding By combining operators Combine them to obtain the current node at time [time]. Combination features of each effective candidate neighbor .
[0097] Generally, the combination operator Vector concatenation is used to make the basic embedding No need to follow Re-encode. The expression is as follows:
[0098] (Equation 6)
[0099] In some embodiments, the combination operator Alternatively, combinations can be made using element-wise multiplication, absolute difference, bilinear mapping, or gating interaction.
[0100] Sub-step 1024: Input the combined features of the effective candidate neighbors into the action evaluation network to obtain the advantage value of each effective candidate neighbor.
[0101] like Figure 2 As shown, the combined features of multiple valid candidate neighbors are... Input Action Evaluation Network Calculate the third intermediate vector according to the following formula. :
[0102] (Equation 7)
[0103] In Equation 7, It is the fifth weighting parameter. It is the fifth bias vector. Then, according to... Calculate the current node at time [time]. The odds of each effective candidate neighbor :
[0104] (Equation 8)
[0105] In Equation 8, It is the sixth weighting parameter. It is the sixth bias vector.
[0106] Advantage value It is a one-dimensional scalar. In some embodiments, all combined features can be stacked into a matrix, and multiple advantage values can be calculated by dimensionality reduction through an action evaluation network.
[0107] For each valid candidate neighbor's combined features All motion evaluation networks use the same parameters. (include , , , (etc.) to perform the calculation.
[0108] Sub-step 1025: Embed the base into the input value head network to obtain the state value of the current node.
[0109] like Figure 2 As shown, in calculating the advantage value of each valid candidate neighbor... At the same time, embed the basic Input Value Head Network First calculate the fourth intermediate vector. :
[0110] (Equation 9)
[0111] In Equation 9, It is the seventh weighting parameter. It is the seventh bias vector.
[0112] Then according to Calculate the state value of the current node :
[0113] (Equation 10)
[0114] In Equation 10, It is the eighth weighting parameter. It is the eighth bias vector.
[0115] State value It is a one-dimensional scalar that represents the overall expected return under the current public routing state, before the specific next hop is distinguished, and does not change with the order of candidate neighbors.
[0116] Step 103: Determine the number of valid candidate neighbors based on the validity mask set, calculate the mean advantage of valid candidate neighbors by combining the advantage value, and calculate the action value of valid candidate neighbors using the state value, advantage value and mean advantage; take the valid candidate neighbor with the largest action value as the next hop for the routing decision.
[0117] The number of valid candidate neighbors can be determined by the set of validity masks, thereby calculating the average advantage of all valid candidate neighbors, and then calculating the action value of all valid candidate neighbors. Subsequently, the action value is used to quantitatively evaluate which candidate neighbor to select as the next hop.
[0118] Optionally, step 103 may include the following sub-steps:
[0119] Sub-step 1031: Add the validity masks in the validity mask set to obtain the number of valid candidate neighbors.
[0120] Validity mask set This indicates the validity of all candidate neighbors. The validity mask in the set can only take values of 0 (representing an invalid candidate neighbor) or 1 (representing a valid candidate neighbor). Therefore, by adding all validity masks in the set, the value of the current node at time t can be obtained. Number of valid candidate neighbors As shown below:
[0121] (Equation 11)
[0122] Sub-step 1032: Sum the advantage values of all valid candidate neighbors and divide by the number of valid candidate neighbors. Take the average of the advantage values to obtain the mean advantage value of the valid candidate neighbors.
[0123] Based on Equation 8, the current node at time [time] can be calculated. mean of the advantages of all valid candidate neighbors :
[0124] (Equation 12)
[0125] Sub-step 1033: Calculate the action value of each valid candidate neighbor based on the current node's state value, the advantage value of each valid candidate neighbor, and the average advantage value.
[0126] like Figure 2 As shown, the value of the current node's state... The advantage value of each effective candidate neighbor and the mean of the advantages Calculate the current node at time [time]. Action value of each valid candidate neighbor As shown below:
[0127] (Equation 13)
[0128] Action value can be quantified as the effect of selecting a candidate neighbor as the next hop after comprehensively considering the overall situation and the unique circumstances of that candidate neighbor.
[0129] Sub-step 1034: Select the valid candidate neighbor with the highest action value as the next hop.
[0130] The above text calculates the action value for each valid candidate neighbor. The valid candidate neighbor with the highest action value can be selected as the next hop for routing decision, that is, the data packet can be sent to the valid candidate neighbor with the highest action value.
[0131] In some embodiments, if the actual engineering interface requires output... indivual( The action value can be determined by setting the action value of invalid candidate neighbors to a preset minimum value according to the validity mask set. , It can be the largest negative value allowed by the numeric type used in the embodiments of this application. For example, the float numeric type can take the largest negative value. .
[0132] In some embodiments, if If the current node has no valid candidate neighbors and all of its candidate neighbors are invalid, the process of calculating the value of the above actions can be skipped. Instead, at least one of the preset processing methods can be triggered, such as waiting, caching, discarding, rolling back routes, or reporting a no-route state.
[0133] The method of this application embodiment will be illustrated below with reference to an exemplary simulation embodiment.
[0134] A low-Earth orbit (LEO) satellite network is constructed, comprising 6 orbital planes with 12 satellites per plane, resulting in a total of 72 satellites and 72 nodes. The orbital altitude is 1200 km, the inclination is 53°, the simulation duration is 600 seconds, and the decision cycle is 1 second. The link capacity between satellites is 100 Mbit / s. Each node has a buffer queue for temporarily storing unforwarded data packets, with a queue capacity of 500 packets. A packet is a data unit processed by the network layer for transmission and routing, and each packet is 1024 bytes long. The total arrival rate (40 Mbit / s) is the sum of newly generated service data injected into the network per unit time at each source node for 20 service flows; this arrival rate does not include forwarding traffic of the same packet by intermediate nodes. Twenty service flows are randomly selected from the source node (the starting node of the data packet transmission) to the destination node. The number of candidate neighbors for each node varies between 0 and 6, influenced by visibility and link status.
[0135] The signal-to-noise ratio should be no less than 3dB, and the bit error rate should be no higher than 10. -5 The remaining bandwidth is no less than 5 Mbit / s, the link duration is no less than 2 seconds, and the target node is reachable. The basic features are set to 16 dimensions, the candidate features of each candidate neighbor (and the effective candidate features of effective candidate neighbors) are 12 dimensions, the basic state encoder is a two-layer fully connected network with dimensions ranging from 16 to 128 to 64, the neighbor encoder is a two-layer fully connected network with dimensions ranging from 12 to 128 to 64, the action evaluation network has a dimension ranging from 128 to 64 to 1, and the value head network has a dimension ranging from 64 to 64 to 1. The action value network is trained using the Adam optimizer with a learning rate of 10. -4 Discount factor 0.99, batch size 256, experience playback capacity 10 5 Training 3×10 5The network is updated every 1000 steps.
[0136] For a given node, set it as the current node, it has 4 candidate neighbors, and the validity mask is... Valid candidate set ,like Figure 3 As shown. The calculated advantage value is... , , Thus, the mean advantage is calculated. Simultaneously calculate the state value of the current node. Then the action values of the three valid candidate neighbors can be calculated. , , ,like Figure 4 As shown, the valid candidate neighbor 1 is selected as the next hop.
[0137] The structured action value network and routing decision device for variable neighbor sets provided in this application embodiment are used to implement Figure 1 The method, wherein the apparatus comprises:
[0138] The feature construction module is used to generate basic features for the current node based on the routing state, construct candidate features for each candidate neighbor of the current node, and divide the candidate neighbors into valid candidate neighbors and invalid candidate neighbors based on the information in the candidate features, thereby generating a set of validity masks describing the validity of the candidate neighbors.
[0139] The encoding and computation module is used to encode the basic features and the candidate features of the effective candidate neighbors using the basic state encoder and the neighbor encoder respectively, to obtain the basic embedding and the effective candidate embedding respectively. The basic embedding and the effective candidate embedding are combined into a combined feature. The action evaluation network is used to calculate the advantage value of the combined feature, and the value head network is used to calculate the state value of the basic embedding.
[0140] The selection module is used to determine the number of valid candidate neighbors based on the set of validity masks, calculate the average advantage of valid candidate neighbors by combining the advantage value, and calculate the action value of valid candidate neighbors using the state value, advantage value and average advantage value; and select the valid candidate neighbor with the largest action value as the next hop for the routing decision.
[0141] The structured action value network and routing decision device for variable neighbor sets in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a GPU box, tablet computer, laptop computer, robot, ultra-mobile personal computer (UMPC), personal digital assistant (PDA), etc., and can also be a server, personal computer (PC), etc. This application embodiment does not specifically limit the specific implementation.
[0142] The structured action value network and routing decision device for variable neighbor sets in this application embodiment can be a device with an operating system. This operating system can be Android, Linux, Windows, or other possible operating systems; this application embodiment does not specifically limit it.
[0143] The structured action value network and routing decision device for variable neighbor sets provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0144] This application provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the structured action value network and routing decision method for variable neighbor sets as described in the previous embodiments.
[0145] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the structured action value network and routing decision method for a variable neighbor set disclosed in this application.
[0146] This application also provides a computer program product that, when run on an electronic device, enables a processor to execute the steps in the structured action value network and routing decision method for a variable neighbor set disclosed in this application.
[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0148] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0151] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.
[0152] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0153] The structured action value network, routing decision method, and apparatus for variable neighbor sets provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application areas based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A structured action value network and routing decision method for variable neighbor sets, characterized in that, The method includes: Based on the routing state, basic features are generated for the current node, and candidate features are constructed for each candidate neighbor of the current node. Based on the information in the candidate features, the candidate neighbors are divided into valid candidate neighbors and invalid candidate neighbors, thereby generating a set of validity masks describing the validity of the candidate neighbors. The basic features and the candidate features of effective candidate neighbors are encoded by the basic state encoder and the neighbor encoder respectively to obtain the basic embedding and the effective candidate embedding. The basic embedding and the effective candidate embedding are combined into a combined feature. The action evaluation network is used to calculate the advantage value of the combined feature, and the value head network is used to calculate the state value of the basic embedding. The number of valid candidate neighbors is determined based on the set of validity masks. The mean advantage of the valid candidate neighbors is calculated by combining the advantage value. The action value of the valid candidate neighbors is calculated using the state value, advantage value, and mean advantage value. The valid candidate neighbor with the highest action value is used as the next hop for the routing decision.
2. The method according to claim 1, characterized in that, Based on the routing state, basic features are generated for the current node, and candidate features are constructed for each candidate neighbor of the current node. Based on the information in the candidate features, candidate neighbors are divided into valid candidate neighbors and invalid candidate neighbors, thereby generating a set of validity masks describing the validity of candidate neighbors, specifically including: Based on the current routing state, generate the basic characteristics of the current node; List the candidate neighbors of the current node, construct candidate features for each candidate neighbor of the current node, and form a candidate feature set from multiple candidate features; Based on the candidate features of each candidate neighbor, valid candidate neighbors and invalid candidate neighbors are determined, and a validity mask is assigned to each valid candidate neighbor and invalid candidate neighbor to generate a set of validity masks.
3. The method according to claim 2, characterized in that, The process involves encoding the basic features and candidate features of effective candidate neighbors using a basic state encoder and a neighbor encoder, respectively, to obtain basic embeddings and effective candidate embeddings. These are then combined into a combined feature. An action evaluation network is used to calculate the combined feature as an advantage value, and a value head network is used to calculate the basic embedding as a state value. Specifically, this includes: The basic features are input into the basic state encoder to obtain the basic embedding; Input the candidate features of the effective candidate neighbors into the neighbor encoder to obtain multiple effective candidate embeddings; Based on the combination operator, the basic embedding and the candidate embeddings of each effective candidate neighbor are combined into multiple combined features; The combined features of effective candidate neighbors are input into the action evaluation network to obtain the advantage value of each effective candidate neighbor. The basic embedded input value head network is used to obtain the state value of the current node.
4. The method according to claim 3, characterized in that, The number of valid candidate neighbors is determined based on the set of validity masks, the mean of the advantage of the valid candidate neighbors is calculated by combining the advantage value, and the action value of the valid candidate neighbors is calculated by using the state value, advantage value and mean advantage. The next hop for routing decisions is determined by using the most valuable candidate neighbor as the next hop, specifically including: The number of valid candidate neighbors is obtained by adding the validity masks in the validity mask set. Add up the dominance values of all valid candidate neighbors and then divide by the number of valid candidate neighbors. Take the average of the dominance values to obtain the mean dominance value of the valid candidate neighbors. Based on the current node's state value, the advantage value of each valid candidate neighbor, and the average advantage value, calculate the action value of each valid candidate neighbor. Choose the nearest valid candidate neighbor with the highest action value as the next hop.
5. A structured action value network and routing decision-making device for a variable neighbor set, used to implement the method described in claims 1-4, characterized in that, The device includes: The feature construction module is used to generate basic features for the current node based on the routing state, construct candidate features for each candidate neighbor of the current node, and divide the candidate neighbors into valid candidate neighbors and invalid candidate neighbors based on the information in the candidate features, thereby generating a set of validity masks describing the validity of the candidate neighbors. The encoding and computation module is used to encode the basic features and the candidate features of the effective candidate neighbors using the basic state encoder and the neighbor encoder respectively, to obtain the basic embedding and the effective candidate embedding respectively. The basic embedding and the effective candidate embedding are combined into a combined feature. The action evaluation network is used to calculate the advantage value of the combined feature, and the value head network is used to calculate the state value of the basic embedding. The selection module is used to determine the number of valid candidate neighbors based on the set of validity masks, calculate the average advantage of valid candidate neighbors by combining the advantage value, and calculate the action value of valid candidate neighbors using the state value, advantage value and average advantage value; and select the valid candidate neighbor with the largest action value as the next hop for the routing decision.
6. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the structured action value network and routing decision method for a variable neighbor set as described in any one of claims 1-4.
7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the structured action value network and routing decision method for a variable neighbor set as described in any one of claims 1-4.