A multicast routing optimization method and system based on constrained discrete maximum entropy reinforcement learning

Abstract

The application discloses a multicast routing optimization method based on a constrained discrete maximum entropy reinforcement learning, first acquires a multicast request, a network topology and a link state, then maps network links in the network topology into edge actions in a discrete action space; generates a legal action mask corresponding to the discrete action space according to a current multicast tree expandable edge set, and identifies whether the edge action belongs to the current multicast tree expandable edge set in the current state through the legal action mask; calls a policy network to process state input, obtains edge action scores corresponding to each edge action, and performs shielding and normalization processing on the edge action scores based on the legal action mask to form a discrete policy distribution constrained by the mask; selects a target edge action based on the discrete policy distribution constrained by the mask, adds an edge corresponding to the target edge action to the current multicast tree; and finally generates a multicast tree and a forwarding path connecting a source node and a plurality of destination nodes step by step.

Description

Technical Field

[0001] This invention relates to the fields of communication networks and multicast routing optimization technology, specifically a multicast routing optimization method and system based on constrained discrete maximum entropy reinforcement learning. Background Technology

[0002] This application relates to the field of multicast routing optimization technology in communication networks, and particularly to a multicast tree construction method based on reinforcement learning. With the rapid development of network infrastructure, the application scenarios of "one-to-many" data transmission in networks are increasing, such as video distribution, IP-based production and broadcasting, and conference transmission. In these scenarios, if traditional unicast methods are used to transmit data, the source node needs to send multiple copies of the data to multiple destination nodes simultaneously, which not only wastes a lot of network resources but also easily leads to network congestion. Multicast technology can achieve more efficient data transmission by transmitting data only once along the same path.

[0003] The core of multicast routing lies in constructing a multicast tree connecting the source node and multiple destination nodes. This problem can be modeled as the Steiner tree problem in graph theory, which is an NP-hard combinatorial optimization problem. Although classic heuristic algorithms such as the KMB algorithm and MPH algorithm can obtain approximate solutions in polynomial time, they lack consideration for the dynamic changes in the global network and are difficult to adapt to network environments with dynamically changing link states.

[0004] In recent years, deep reinforcement learning has been gradually introduced into the field of multicast routing optimization because it can learn optimal policies through interaction with the environment. Existing methods typically map links or nodes in the network topology into an action space, construct the current multicast tree state and link states (such as bandwidth, latency, and packet loss rate) as state inputs, and gradually select actions through a policy network to expand the multicast tree.

[0005] However, existing multicast routing methods based on deep reinforcement learning have the following drawbacks when dealing with the action space: First, some methods (such as DRL-M4MR) treat all links in the network as the agent's action space. However, because the expansion of the multicast tree is strongly state-dependent, not all links in the network topology are suitable as the next expansion action in the current state—for example, edges already added to the multicast tree, edges not adjacent to the current multicast tree, and edges that would form a loop after being added are not suitable as executable actions for the current step. When the agent samples from the action space comprised of all links, it is easy to select a large number of invalid actions (negative samples), leading to slow convergence and low training efficiency.

[0006] Secondly, although some improved methods reduce the action space from the entire link to the neighboring edges of the current node through hierarchical reinforcement learning, thus reducing the sampling opportunities of invalid actions, they still do not solve the problem of training target computation: illegal actions may still participate in the computation of policy loss, entropy term and target value estimation, thereby polluting the training target with invalid action information and affecting the update direction of the policy decision model.

[0007] Third, if actions are filtered only during the inference phase, while the training phase still calculates the training objective based on the unconstrained action space, the action space faced during training will be inconsistent with the action space used during deployment and execution, which may lead to policy transfer bias and affect the stability of the multicast tree generation process.

[0008] Therefore, this application aims to provide a multicast routing optimization method and system, which can generate a legal action mask based on the current multicast tree expandable edge set. This mask is used not only for filtering in the action selection stage, but also for constraining the current state policy distribution, policy loss, next state policy distribution, entropy term, and target value estimation, thereby reducing the interference of non-expandable edges on policy sampling and value estimation, and keeping the training action space consistent with the inference execution space. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and propose a multicast routing optimization method and system based on constrained discrete maximum entropy reinforcement learning. This method addresses how to reuse the same legal action mask generation rules in the training and inference phases, thereby improving the consistency between the training action space and the inference execution space.

[0010] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows: A multicast routing optimization method based on constrained discrete maximum entropy reinforcement learning includes: S101. Obtain a multicast request, wherein the multicast request includes a source node and multiple destination nodes; S102. Obtain network topology and link status, construct a current multicast tree state matrix, a normalized bandwidth matrix, a normalized delay matrix, and a normalized packet loss rate matrix based on the network topology and link status, and construct a state input based on the current multicast tree state matrix, the normalized bandwidth matrix, the normalized delay matrix, and the normalized packet loss rate matrix. S103. Map the network links or edges in the network topology to edge actions in the discrete action space; S104. Generate a valid action mask corresponding to the discrete action space based on the current multicast tree expandable edge set. The valid action mask is used to identify whether the edge action belongs to the current multicast tree expandable edge set in the current state. S105. The policy network is invoked to process the state input to obtain the edge action score, and the edge action score is masked and normalized based on the legal action mask to form a discrete policy distribution constrained by the mask. S106. Select edge actions based on the mask-constrained discrete strategy distribution, add the edge corresponding to the selected edge action to the current multicast tree, so as to expand the current multicast tree and update the current multicast tree state. S107. During the training phase, the policy distribution and policy loss of the current state are constrained based on the legal action mask of the current state, and the policy distribution, entropy term and target value estimate of the next state are constrained based on the legal action mask of the next state, so as to update the policy decision model. S108. In the inference phase, the same legal action mask generation rules as in the training phase are reused to gradually generate a multicast tree or forwarding path connecting the source node and the multiple destination nodes.

[0011] Preferably, the state input includes a four-channel matrix, which is formed by concatenating the current multicast tree state matrix, the normalized bandwidth matrix, the normalized delay matrix, and the normalized packet loss rate matrix according to the channel dimension.

[0012] Preferably, the discrete action space consists of all network links or edges in the network topology, each edge action corresponds to a candidate edge, and there is a one-to-one mapping relationship between the edge action and the candidate edge.

[0013] Preferably, a legal action mask corresponding to the discrete action space is generated based on the current multicast tree expandable edge set, including: Obtain the set of nodes and the set of edges already contained in the current multicast tree; Determine the boundary nodes of the current multicast tree; Obtain candidate edges adjacent to the boundary node; Remove duplicate edges entering the tree, non-adjacent expansion edges, and edges that cause the current multicast tree to form an invalid expansion from the candidate edges to obtain the set of expandable edges of the current multicast tree; Actions that belong to the current multicast tree expandable edge set are marked as optional, and actions that do not belong to the current multicast tree expandable edge set are marked as non-optional.

[0014] Preferably, excluding edges that cause invalid expansion of the current multicast tree includes: performing an acyclicity determination based on the node set of the current multicast tree and the two endpoints of the candidate edge; if the addition of the candidate edge causes the current multicast tree to form a cycle, then the edge action corresponding to the candidate edge is marked as unselectable.

[0015] Preferably, the mask-constrained discrete strategy distribution is determined in the following manner: when hour, ; when hour, ; Where s represents the state input, This represents the action of the i-th edge. This represents the edge action score corresponding to the i-th edge action. This represents the mask value corresponding to the i-th edge action in the legal action mask, and the summation range is the set of edge actions marked as optional by the legal action mask.

[0016] Preferably, the strategy decision-making model adopts the maximum entropy discrete Actor-Critic model, and the training objective of the maximum entropy discrete Actor-Critic model includes the expected reward term and the strategy entropy term.

[0017] Preferably, the strategy decision model adopts a discrete SAC-BBF or BBF training paradigm, wherein the training paradigm includes at least one of configurable update frequency, multiple parameter update mechanism, entropy regularization and temperature coefficient update.

[0018] Preferably, the strategy decision model includes a first Q-network, a second Q-network, and a target Q-network, wherein the target Q-network is updated based on the parameters of the first Q-network or the second Q-network according to a soft update coefficient.

[0019] Preferably, during the training phase, the current state, the selected edge action, the reward, the next state, the termination flag, the mask of the legal action in the current state, and the mask of the legal action in the next state are stored as experience samples in the experience cache, and the policy decision model is trained by sampling from the experience cache.

[0020] Preferably, the reward or evaluation metric includes at least one of the following: multicast tree construction completion rate, bandwidth, latency, packet loss rate, and multicast tree length.

[0021] Preferably, multiple candidate strategy decision models or multiple candidate multicast trees are evaluated based on the multicast tree construction completion rate, bandwidth, latency, packet loss rate and multicast tree length, and the strategy decision model or multicast tree used for deployment is selected according to the evaluation results.

[0022] Preferably, the multicast routing optimization method is applied to traditional multicast, IP-based production and broadcasting, video distribution, or conference transmission scenarios.

[0023] This invention also provides a multicast routing optimization system based on constrained discrete maximum entropy reinforcement learning, comprising: The request acquisition module is used to acquire multicast requests, wherein the multicast requests include a source node and multiple destination nodes; The state construction module is used to obtain network topology and link status, and construct the current multicast tree state matrix, normalized bandwidth matrix, normalized delay matrix and normalized packet loss rate matrix based on the network topology and link status. The action space management module is used to map network links or edges in the network topology to edge actions in the discrete action space. The mask generation module is used to generate a legal action mask corresponding to the discrete action space based on the current multicast tree expandable edge set; The policy distribution generation module is used to call the policy network to obtain edge action scores and form a discrete policy distribution constrained by the mask based on the legal action mask; The multicast tree expansion module is used to select edge actions and expand the current multicast tree according to the mask-constrained discrete strategy distribution; The model training module is used to constrain the current state policy distribution and policy loss based on the current state legal action mask during the training phase, and to constrain the next state policy distribution, entropy term and target value estimation based on the next state legal action mask. The inference deployment module is used to reuse the same legal action mask generation rules as those in the training phase during the inference phase to generate multicast trees or forwarding paths.

[0024] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned multicast routing optimization method based on constrained discrete maximum entropy reinforcement learning.

[0025] This invention has the following characteristics and beneficial effects: First, generating a legal action mask based on the current set of scalable edges in the multicast tree helps reduce the chances of non-scalable edges participating in policy sampling and multicast tree expansion.

[0026] Second, using the current state legal action mask in the current state policy distribution and policy loss, and using the next state legal action mask in the next state policy distribution, entropy term and target value estimation, helps to keep the target value estimation consistent with the actual set of executable edges in the corresponding state.

[0027] Third, reusing the same legal action mask generation rules in the training and inference phases helps improve the consistency between the training action space and the inference execution space.

[0028] Fourth, in a preferred implementation of maximum entropy discrete policy learning, policy exploration can focus on the current set of executable edges, which helps to match the training process with the constraint characteristics of the progressive expansion of the multicast tree.

[0029] The above-mentioned effects are technical effects that can be obtained based on the technical solution of this application, and do not represent the same level of performance improvement under all network topologies or all training configurations. Attached Figure Description

[0030] Figure 1 The flowchart illustrates a multicast routing optimization method based on constrained discrete maximum entropy reinforcement learning, as provided in this embodiment of the invention.

[0031] Figure 2 This is a diagram of the multicast routing optimization system architecture provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the mask constraint training and update process provided in an embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0033] Example 1 This embodiment provides a multicast routing optimization method based on constrained discrete maximum entropy reinforcement learning. This method takes network topology and link state information as input, and constructs a multicast tree by progressively selecting edge actions in a discrete action space through a policy network. During training and inference, a valid action mask is used to constrain the policy distribution, ultimately outputting the optimal multicast tree and forwarding path connecting the source node and multiple destination nodes. Figure 1 As shown, this embodiment includes the following steps.

[0034] S101. Obtain a multicast request, wherein the multicast request includes a source node and multiple destination nodes.

[0035] In this embodiment, the source node is used to send multicast data, and the multiple destination nodes are used to receive the same multicast data. The multicast request may also include a network topology instance identifier, a service instance identifier, or other identifiers used to locate the network environment.

[0036] In data flow, after a multicast request enters the request acquisition module, it is converted into node set parameters for subsequent routing decisions. These node set parameters include at least a source node identifier and a destination node identifier set. The state construction module can determine the initialization state of the current multicast tree based on these node set parameters, for example, using the source node as the initial node of the current multicast tree.

[0037] If a multicast request lacks a source node or destination node set, the system can return a request exception message or wait for the upper-layer control entity to complete the multicast request. If the destination node set is empty, the system can terminate the current multicast tree construction process. The above exception handling does not change the core process of the distribution and training update of the legal action mask constraint strategy in this application.

[0038] It should be noted that, in one optional application scenario, the multicast request can originate from IP-based production and broadcasting services, video distribution services, or conferencing transmission services. This application scenario only indicates the source of the request and does not change the multicast tree construction process based on the source node, multiple destination nodes, and network topology.

[0039] S102. Obtain network topology and link status. Construct a four-channel matrix based on the network topology and link status, and use it as the status input. Obtain the network topology of the target network And link status information, the network topology includes a set of nodes. Sum of edges The link status information includes the remaining bandwidth of each link. Transmission delay Packet loss rate and error rate ,in This represents the two endpoint nodes of the link.

[0040] A four-channel matrix is ​​constructed based on the network topology and link status, serving as the state input. Specifically, the current multicast tree state matrix is ​​constructed. Normalized bandwidth matrix Normalized time delay matrix and normalized packet loss rate matrix The four matrices are concatenated according to the channel dimension to obtain a four-channel state matrix. .

[0041] Among them, the current multicast tree state matrix Rows and columns correspond to network nodes; if an edge has been added to the current multicast tree, the first identifier is written to the corresponding position in the matrix; if an edge has not been added to the current multicast tree, the second identifier is written. Source and destination nodes can be represented by appended vectors, matrix diagonal identifiers, or node attribute encodings.

[0042] Furthermore, all four matrices are N×N matrices corresponding to the number of network nodes, with rows and columns corresponding to network node numbers; This indicates that the edge between node i and node j has been added to the current multicast tree; This indicates that the edge between node i and node j has not yet been added to the current multicast tree.

[0043] , represents the normalized value of the link bandwidth between node i and node j; This indicates that there is no link between node i and node j.

[0044] and Similarly, these correspond to latency and packet loss rate respectively. The four-channel state matrix All possible values ​​constitute the state space. ,in The number of states.

[0045] in, This indicates concatenation along the channel dimension. The state input is sent to the policy network and the Q network to generate edge action scores and value estimates.

[0046] If network topology or link status is missing, the status construction module can construct status input using the most recent valid data collection results, preset link status, or missing data flags. If the missing data cannot meet the input requirements of the policy decision model, the system can terminate the routing generation process for the current multicast request.

[0047] S103. Map the network links in the network topology to edge actions in the discrete action space. Map all network links in the network topology to a discrete action space. The discrete action space consists of all network links in the network topology, and each edge action... With a candidate edge One-to-one mapping. When an edge takes an action... When selected, the candidate edge corresponding to the action of that edge. It is added to the current multicast tree.

[0048] Suppose that there are a total of If there are 1 link, then the size of the discrete action space is 1. The action space can be represented as Each edge action A unique candidate edge .

[0049] The action space management module maintains the mapping relationship between edge actions and candidate edges so that after the policy network outputs the edge action score, the selected edge action is converted into the current multicast tree expansion operation.

[0050] In data flow, the dimension of the discrete action space is consistent with the dimension of the policy network output layer. That is, the policy network outputs an edge action score for each edge action. The legal action mask is also consistent with the dimension of the discrete action space, thus allowing each edge action to be identified as optional.

[0051] Understandably, edge actions can also be replaced with next-hop actions, candidate path segment actions, or other discrete actions that can be mapped to multicast tree expansion operations. As long as the actions can generate a valid action mask based on the current multicast tree's expandable edge set or expandable action set, the technical solution of this application can be applied.

[0052] S104. Generate a valid action mask corresponding to the discrete action space based on the current multicast tree expandable edge set, and use the valid action mask to identify whether the edge action belongs to the current multicast tree expandable edge set in the current state.

[0053] The mask generation module determines the set of expandable edges for the current multicast tree based on the current multicast tree state. Specifically, the mask generation module obtains the set of nodes and edges already included in the current multicast tree, determines the boundary nodes of the current multicast tree, obtains candidate edges adjacent to the boundary nodes, and excludes duplicate edges and non-adjacent expandable edges from the candidate edges.

[0054] In a further implementation, the mask generation module also performs an acyclicity check. If adding a candidate edge would cause the current multicast tree to form a cycle, then that candidate edge is not included in the current multicast tree expandable edge set. In this way, the current multicast tree expandable edge set can reflect the actual edge expansion operations that can be performed in the current state.

[0055] Specifically, at each time step of the current multicast tree construction, a valid action mask corresponding to the discrete action space is generated based on the current multicast tree's expandable edge set. The legal action mask is a length equal to A binary vector is used to identify whether each edge action belongs to the current multicast tree expandable edge set in the current state.

[0056] The mask generation module generates a valid action mask based on the current multicast tree expandable edge set. For the i-th edge action ,like If the corresponding candidate edge belongs to the current multicast tree's expandable edge set, then... Select optional identifier; if If the corresponding candidate edge does not belong to the current multicast tree's expandable edge set, then... Selectable / unselectable flag.

[0057] If the current multicast tree's expandable edge set is empty, the system can determine that the current multicast tree expansion process cannot continue and output a construction failure flag, a partial multicast tree, or an abnormal state. Alternatively, in optional implementations, the link state can be reacquired or the candidate path search can be reinitialized. The above anomaly handling does not change the constraint relationship between the legal action mask and the policy distribution and training updates.

[0058] The legal action mask is generated in the following way: (1) Get the current multicast tree The set of nodes already included Sum of edges .

[0059] (2) Determine the set of boundary nodes of the current multicast tree. This refers to the nodes in the current multicast tree that are connected to edges that are not yet included in the current multicast tree.

[0060] (3) Obtain the set of candidate edges adjacent to the boundary nodes. .

[0061] (4) Remove from the candidate edges edges that have already been added to the current multicast tree, extended edges that are not adjacent to the current multicast tree, and edges that, after being added, cause the current multicast tree to form an invalid extension. Edges that, after being added, cause the current multicast tree to form an invalid extension include edges that, after being added, create a cycle in the current multicast tree. Based on the set of nodes already included in the current multicast tree. Perform an acyclicity check on the two endpoints of the candidate edge: if the candidate edge One endpoint It already belongs to the set of already included nodes. And the other end It also belongs to the set of already included nodes. If the addition of the candidate edge would cause a loop in the current multicast tree, the edge action corresponding to the candidate edge would be marked as unselectable.

[0062] (5) Expand the set of edges belonging to the current multicast tree. Edge actions are marked as optional (e.g., mask value is 1), and edge actions that do not belong to the current multicast tree's expandable edge set are marked as non-optional (e.g., mask value is 0). That is, for each edge action... Its corresponding legal action mask value satisfy:

[0063] S105. The policy network is invoked to process the state input, obtain the edge action score corresponding to each edge action, and perform masking and normalization processing on the edge action score based on the legal action mask to form a mask-constrained discrete policy distribution. Among them, the edge action marked as unselectable by the legal action mask has a probability of zero in the mask-constrained discrete policy distribution.

[0064] The policy distribution generation module uses the state input as input to the policy network to obtain the edge action score corresponding to each edge action. Subsequently, the policy distribution generation module uses the legal action mask as a basis. The edge action scores are masked and normalized.

[0065] Specifically, the call strategy network For the current state (i.e., four-channel state matrix) The policy network processes the data and outputs edge action scores corresponding to each edge action. ,in The mapping function representing the policy network, This represents network parameters.

[0066] The edge action scores are masked and normalized based on the legal action mask to form a discrete policy distribution constrained by the mask. The mask-constrained discrete strategy distribution is determined as follows: For each edge action If the legal action mask value corresponding to the action on that side Then the probability of the mask-constrained strategy for the action on that side is:

[0067] If the mask value corresponding to the action on this side is a valid action mask value (Without optional identifiers), the probability of the mask-constrained strategy for this edge action is:

[0068] In this way, the probability of an edge action marked as unselectable by the legal action mask is zero in the masked discrete policy distribution, ensuring that the policy network selects actions only from legal scalable edges.

[0069] in, This indicates the state input. This represents the action of the i-th edge. This represents the edge action score corresponding to the i-th edge action. This represents the mask value corresponding to the i-th edge action in the legal action mask, and the summation range is the set of edge actions marked as optional by the legal action mask.

[0070] The summation range only includes edge actions whose legal action mask indicates they are optional. Through this process, non-scalable edge actions are not included in the normalization calculation of the current policy distribution, and their probability in the current policy distribution is zero.

[0071] In data flow, the mask-constrained discrete policy distribution is sent to the multicast tree expansion module. During the training phase, the discrete policy distribution is also sent to the model training module for policy loss calculation. If all edge actions are marked as unselectable, the policy distribution generation module does not perform normalization processing and returns a non-scalable state to the upstream module.

[0072] S106. Select the target edge action based on the discrete strategy distribution constrained by the mask, add the edge corresponding to the target edge action to the current multicast tree, so as to expand the current multicast tree and update the current multicast tree state.

[0073] Specifically, the multicast tree expansion module selects edge actions based on the mask-constrained discrete policy distribution. During the training phase, the system can sample edge actions according to the discrete policy distribution; during the inference phase, the system can select the edge action with the highest probability or the edge action with the highest score and that is valid.

[0074] After selecting an edge action, the multicast tree expansion module queries the mapping relationship in the action space management module to determine the candidate edge corresponding to the edge action and adds the candidate edge to the current multicast tree. Subsequently, the state construction module updates the current multicast tree state matrix, and the mask generation module updates the current multicast tree expandable edge set and the legal action mask.

[0075] If the addition of a candidate edge connects a new destination node, the system updates the destination node coverage status. If all destination nodes have been connected, the system determines that the current multicast tree construction is complete. If a preset termination condition is met but not all destination nodes are connected, the system can output the current candidate multicast tree and the termination status.

[0076] In this embodiment, the discrete strategy distribution is based on the mask constraint. Select target side action In one specific implementation, random sampling is performed according to the mask-constrained discrete policy distribution. In another specific implementation, the edge action with the highest probability is directly selected.

[0077] Action on the target side Corresponding edges Add to current multicast tree This expands the current multicast tree and updates its state, including updating the set of nodes already included. The set of edges already included and multicast tree state matrix .

[0078] Repeat steps 104 to 106 until the current multicast tree connects the source node and all destination nodes, or the current multicast tree's scalable edge set is empty (indicating that a valid multicast tree cannot be constructed).

[0079] S107. Construct and train the policy decision model.

[0080] The policy decision model is a maximum entropy discrete Actor-Critic model. In one specific implementation, the policy decision model is constructed and trained using a discrete SAC (Soft Actor-Critic) training paradigm or a BBF (Bootstrap with Bootstrapping and Fusion) training paradigm.

[0081] The strategy decision-making model includes a first Q network. Second Q network and target Q network , The first Q-network and the second Q-network are used to estimate the value of the current state-action pair, and the target Q-network is updated using a soft update method to stabilize the training process.

[0082] Specifically, the first Q-network, the second Q-network, and the target Q-network constitute the value assessment part, or Critic part, in the maximum entropy discrete Actor-Critic policy decision model; the policy network above is the Actor part. More specifically, the current model can be understood as: Policy Network: Receives the current state and the mask of legal actions, outputs the action score of each action, and forms a discrete policy distribution constrained by the mask; First Q network and second Q network: These belong to the current value assessment network and are used to estimate the action value of each legal edge action in the current state. The double Q structure is used to reduce the risk of overestimation of value. Target Q-network: This is a target value evaluation network. It is a slow-updated copy of the current Q-network and is used to calculate the target value during training, making the training process more stable.

[0083] Therefore, the three types of Q-networks are not three additional models independent of Actor-Critic, but rather components of the Critic side of the maximum entropy discrete Actor-Critic model.

[0084] The soft update method is as follows: the parameters of the target Q-network are updated by multiplying the current Q-network parameters by a soft update coefficient, plus the result of multiplying the current parameters of the target Q-network by one and subtracting the soft update coefficient. Taking the first Q-network as an example, its target network parameter update method is as follows:

[0085] in, This represents the soft update coefficient, which is usually a small value. In this embodiment, it is adopted. , This represents the parameters of the current first Q-network. This represents the parameters of the target first Q-network. The target network update method for the second Q-network is the same.

[0086] The training objectives of the maximum entropy discrete Actor-Critic model include the expected reward term and the policy entropy term. Specifically: The model training module stores the interaction data from the training phase as experience samples. These experience samples include the current state, the selected edge action, the reward, the next state, the termination flag, the mask for legal actions in the current state, and the mask for legal actions in the next state. This experience sample can be represented as:

[0087] in, Indicates the current state. Indicates the selected edge action. Indicates a reward. Indicates the next state, and done_t indicates whether termination has occurred. This represents the mask for legal actions in the current state. This represents the mask for the legal action in the next state.

[0088] For the current state The model training module uses the current state's legal action mask. Form a policy distribution for the current state that is constrained by a mask, and sum only the set of executable edges for the current state in the policy loss calculation.

[0089] Strategy Loss Let $\frac{ ...

[0090] in, Represents the entropy weighting coefficient. Indicates the first A Q network pair of states and actions Value estimate.

[0091] For the next state The model training module uses the next-state legal action mask. A mask-constrained policy distribution for the next state is formed, and the entropy term and target value estimate are calculated based on this distribution. The target value estimate can be expressed as:

[0092] The range of summation is only... The side action.

[0093] The target Q value can be expressed as:

[0094] in, This represents the discount factor. The model training module updates the Q-network based on the target Q-value and the policy network based on the policy loss.

[0095] If the next state is a termination state, the model training module may not calculate the target value of the next state, or it may set the target value corresponding to the termination state to a preset termination value. By saving and using the mask of legal actions in the next state, the training process can limit the target value estimation to the set of actually executable edges in the next state.

[0096] During the training phase, the current state will be... The side action selected in the current state ,award Next state Termination Mark The legal action mask corresponding to the current state The legal action mask corresponding to the next state Stored as an experience sample in the experience cache. During training, the experience samples are sampled from the experience buffer. The current state policy distribution and the policy loss are constrained by the current state legal action mask in the experience samples. The next state policy distribution, the entropy term, and the target value estimate are constrained by the next state legal action mask in the experience samples.

[0097] Specifically, the first Q-network and the second Q-network use the current state and the current action to output the value of the current state and action; The next state's legal action mask and the next state are used to form the masked policy distribution of the next state; The target Q-network combines the masked policy distribution of the next state to calculate the target value estimate; The target value estimation and reward together constitute the training objective of the Q network; The first Q network and the second Q network update their parameters according to the training objective, respectively. The current state's legal action mask is used to constrain the current state's policy distribution and policy loss, thereby updating the policy network.

[0098] The training paradigm includes at least one of the following mechanisms: configurable policy network update frequency, multiple parameter update mechanism after each round of environment interaction, entropy regularization mechanism, and adaptive update mechanism for temperature coefficient.

[0099] In one specific implementation, the reward function of the policy decision model is constructed based on at least one of the following evaluation metrics: multicast tree construction completion rate, link bandwidth, link latency, link packet loss rate, and multicast tree length. For example, when the agent performs an action... State Transform into a state If the added edge contributes a new destination node to the multicast tree, a positive reward is given; if the added edge causes the multicast tree to form a cycle or exceed resource constraints, a negative reward (penalty) is given.

[0100] S108. During the inference phase, the legal action mask generation rules of the trained policy decision model are used to gradually generate a multicast tree connecting the source node and multiple destination nodes, as well as forwarding paths.

[0101] During the inference phase, the inference deployment module loads the trained policy decision model and reuses the same legal action mask generation rules from the training phase. For each inference step, the system performs state input construction, legal action mask generation, edge action score calculation, mask-constrained discrete policy distribution formation, and edge action selection.

[0102] Because the same legal action mask generation rules are used in both the inference and training phases, the range of action choices learned by the model during training remains consistent with the range of action choices executed during deployment. This consistency helps reduce the discrepancy between the training action space and the inference execution space.

[0103] Once all destination nodes are connected, the inference deployment module outputs a multicast tree or forwarding path. The control deployment entity can then convert this multicast tree or forwarding path into deployable forwarding table entries and provide them to the network control entity for execution.

[0104] Example 2 The difference between this embodiment and Embodiment 1 is that the multicast routing optimization method is applied to IP-based production and broadcasting scenarios.

[0105] In IP-based production and broadcasting scenarios, multiple receiving ends need to simultaneously receive video stream data from the same source node. The process involves acquiring multicast requests from the source node (video server) and multiple destination nodes (each receiving terminal), obtaining the topology and link status of the production and broadcasting network, and constructing a multicast tree connecting the source node and all destination nodes step-by-step according to the method described in Example 1. The forwarding paths corresponding to the generated multicast tree are then configured in the production and broadcasting network switches to achieve efficient multicast distribution of video stream data.

[0106] Example 3 The difference between this embodiment and Embodiment 1 is that the multicast routing optimization method is applied to a video distribution scenario.

[0107] In video distribution scenarios, edge nodes of a Content Delivery Network (CDN) need to simultaneously send video content to multiple cache nodes. By constructing an optimal multicast tree according to the method described in Example 1, the overall transmission cost of the multicast tree is minimized while ensuring link bandwidth and latency constraints, thus achieving efficient distribution of video content.

[0108] Example 4 The difference between this embodiment and Embodiment 1 is that the multicast routing optimization method is applied to a conference transmission scenario.

[0109] In multi-party video conferencing scenarios, the conference server needs to send audio and video data to multiple participating terminals simultaneously. A multicast tree is constructed according to the method described in Example 1, and the multicast tree structure is dynamically updated in real time to respond to the dynamic joining and leaving of participating terminals, ensuring the real-time transmission of conference data.

[0110] Example 5 This embodiment provides a multicast routing optimization system based on constrained discrete maximum entropy reinforcement learning, including: The request acquisition module is used to acquire multicast requests, which include a source node and multiple destination nodes.

[0111] The state construction module is used to obtain network topology and link status, construct a current multicast tree state matrix, a normalized bandwidth matrix, a normalized delay matrix, and a normalized packet loss rate matrix based on the network topology and link status, and construct state input based on the current multicast tree state matrix, the normalized bandwidth matrix, the normalized delay matrix, and the normalized packet loss rate matrix.

[0112] The action space management module is used to map network links in the network topology to edge actions in the discrete action space.

[0113] The mask generation module is used to generate a legal action mask corresponding to the discrete action space based on the current multicast tree expandable edge set.

[0114] The policy distribution generation module is used to call the policy network to process the state input to obtain the edge action score corresponding to each edge action, and to perform masking and normalization processing on the edge action score based on the legal action mask to form a discrete policy distribution constrained by the mask.

[0115] The multicast tree expansion module is used to select target edge actions based on the mask-constrained discrete strategy distribution, add the edges corresponding to the target edge actions to the current multicast tree to expand the current multicast tree, and update the current multicast tree state.

[0116] The model training module is used to constrain the current state policy distribution based on the current state legal action mask and calculate the policy loss during the training phase, and to constrain the next state policy distribution based on the next state legal action mask and calculate the entropy term and target value estimate respectively, so as to update the policy decision model.

[0117] The inference deployment module is used to generate a multicast tree or forwarding path connecting the source node and the multiple destination nodes step by step during the inference phase, using the same legal action mask generation rules as those in the training phase.

[0118] The specific implementation methods of each of the above modules are described in the corresponding steps in Example 1, and will not be repeated here.

[0119] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multicast routing optimization method based on constrained discrete maximum entropy reinforcement learning, characterized in that, include: Acquire multicast requests, network topology, and link status, wherein the multicast requests include a source node and multiple destination nodes; A four-channel matrix is ​​constructed based on network topology and link status, and used as the status input; Map the network links in the network topology to edge actions in the discrete action space; Generate a valid action mask corresponding to the discrete action space based on the current multicast tree expandable edge set, and use the valid action mask to identify whether the edge action belongs to the current multicast tree expandable edge set in the current state; The policy network is invoked to process the state input, obtain the edge action score corresponding to each edge action, and mask and normalize the edge action score based on the legal action mask to form a mask-constrained discrete policy distribution. Among them, the edge action marked as unselectable by the legal action mask has a probability of zero in the mask-constrained discrete policy distribution. The target edge action is selected based on the discrete strategy distribution constrained by the mask, and the edge corresponding to the target edge action is added to the current multicast tree to expand the current multicast tree and update the current multicast tree state. A policy decision model is constructed and trained, the policy decision model including a first Q network, a second Q network and a target Q network, the target Q network being updated in a soft update manner; During the inference phase, the legal action mask generation rules of the trained policy decision model are used to gradually generate a multicast tree connecting the source node and multiple destination nodes, as well as forwarding paths.

2. The multicast routing optimization method according to claim 1, characterized in that, Based on the network topology and link status, a current multicast tree state matrix, a normalized bandwidth matrix, a normalized delay matrix, and a normalized packet loss rate matrix are constructed. The four-channel matrix is ​​formed by concatenating the current multicast tree state matrix, the normalized bandwidth matrix, the normalized delay matrix, and the normalized packet loss rate matrix according to the channel dimension.

3. The multicast routing optimization method according to claim 1, characterized in that, The discrete action space consists of all network links in the network topology. Each action is mapped one-to-one with a candidate edge. When an action is selected, the candidate edge corresponding to the action is added to the current multicast tree.

4. The multicast routing optimization method according to claim 1, characterized in that, Generate a valid action mask corresponding to the discrete action space based on the current multicast tree expandable edge set, including: Obtain the set of nodes and the set of edges already contained in the current multicast tree; Determine the boundary nodes of the current multicast tree, where the boundary nodes are the nodes already included in the current multicast tree that are connected to the edges of nodes not yet included in the current multicast tree; Obtain candidate edges adjacent to the boundary nodes; Remove from the candidate edges edges that have been added to the current multicast tree, extended edges that are not adjacent to the current multicast tree, and edges that, when added, would cause the current multicast tree to form an invalid extension; Mark edge actions that belong to the current multicast tree's expandable edge set as optional, and mark edge actions that do not belong to the current multicast tree's expandable edge set as non-optional.

5. The multicast routing optimization method according to claim 4, characterized in that, Adding an edge that causes an invalid expansion of the current multicast tree includes adding an edge that causes a cycle in the current multicast tree. Excluding edges that cause a cycle in the current multicast tree specifically involves: performing an acyclicity check based on the set of nodes already included in the current multicast tree and the two endpoints of the candidate edge; if one endpoint of the candidate edge already belongs to the set of nodes already included and the other endpoint also belongs to the set of nodes already included, then it is determined that adding the candidate edge will cause a cycle in the current multicast tree, and the edge action corresponding to the candidate edge is marked as unselectable.

6. The multicast routing optimization method according to claim 1, characterized in that, The mask-constrained discrete strategy distribution is determined as follows: For each edge action, if the legal action mask value corresponding to the edge action is optional, then the probability of the mask-constrained policy of the edge action is the natural exponent value of the edge action score of the edge action divided by the sum of the natural exponent values ​​of the edge action scores corresponding to all edge actions whose legal action mask values ​​are optional. If the mask value of the legal action corresponding to the edge action is a non-selectable flag, then the probability of the mask-constrained strategy for the edge action is zero.

7. The multicast routing optimization method according to claim 1, characterized in that, The policy decision model is a maximum entropy discrete Actor-Critic model. The training objective of the maximum entropy discrete Actor-Critic model includes an expected reward term and an entropy term. The policy loss of the updated policy decision model is the expected value under the constrained current state policy distribution. Each term in the expected value is the smaller value of the entropy weight multiplied by the logarithm of the constrained current state policy distribution minus the current state action value. The entropy term is the sum of the negative values ​​of the product of the constrained next state policy distribution and its logarithm. The target value estimate is the expected value under the constrained next state policy distribution. Each term in the expected value is the smaller value of the next state action minus the entropy weight multiplied by the logarithm of the constrained next state policy distribution.

8. The multicast routing optimization method according to claim 7, characterized in that, The policy decision model adopts a discrete SAC-BBF training paradigm or a BBF training paradigm. The training paradigm includes at least one of the following mechanisms: configurable policy network update frequency, multiple parameter update mechanism after each round of environmental interaction, entropy regularization mechanism, and adaptive update mechanism for temperature coefficient.

9. The multicast routing optimization method according to claim 7, characterized in that, The soft update method is as follows: the parameters of the target Q network are updated to the current Q network parameters multiplied by the soft update coefficient, plus the current parameters of the target Q network multiplied by one minus the soft update coefficient; the current Q network is either the first Q network or the second Q network.

10. The multicast routing optimization method according to claim 7, characterized in that, During the training phase, the current state, the edge action selected in the current state, the reward, the next state, the termination flag, the legal action mask corresponding to the current state, and the legal action mask corresponding to the next state are stored as an experience sample in the experience buffer. During training, the experience sample is sampled from the experience buffer. The legal action mask of the current state in the experience sample is used to constrain the policy distribution of the current state and the policy loss. The legal action mask of the next state in the experience sample is used to constrain the policy distribution of the next state, the entropy term, and the target value estimate.

11. The multicast routing optimization method according to claim 1, characterized in that, The reward function of the strategy decision model is constructed based on at least one of the following evaluation metrics: multicast tree construction completion rate, link bandwidth, link latency, link packet loss rate, and multicast tree length.

12. The multicast routing optimization method according to claim 11, characterized in that, It also includes a model evaluation and selection step: based on at least one of the following indicators, namely the multicast tree construction completion rate, the bandwidth, the latency, the packet loss rate and the multicast tree length, a quantitative evaluation is performed on multiple candidate strategy decision models or multiple candidate multicast trees, and a strategy decision model for deployment is selected from the multiple candidate strategy decision models or a multicast tree for deployment is selected from the multiple candidate multicast trees according to the evaluation results.

13. The multicast routing optimization method according to claim 1, characterized in that, The multicast routing optimization method is applicable to at least one of the following scenarios: traditional IP multicast scenarios, IP-based production and broadcasting scenarios, video distribution scenarios, and conference transmission scenarios.

14. A multicast routing optimization system based on constrained discrete maximum entropy reinforcement learning, characterized in that, include: The request retrieval module is used to retrieve multicast requests; The state construction module is used to obtain network topology and link status, and construct state input based on the network topology and link status; The action space management module is used to map network links in the network topology to edge actions in the discrete action space; The mask generation module is used to generate a legal action mask corresponding to the discrete action space based on the current multicast tree expandable edge set; The policy distribution generation module is used to call the policy network to process the state input to obtain the edge action score corresponding to each edge action, and to constrain the edge action score based on the legal action mask to form a discrete policy distribution constrained by the mask. The multicast tree expansion module is used to select target edge actions based on the mask-constrained discrete strategy distribution, add the edges corresponding to the target edge actions to the current multicast tree to expand the current multicast tree, and update the current multicast tree state. The model training module is used to update the parameters of the policy network based on the maximum entropy reinforcement learning objective function and the policy decision model by using the legal action mask to constrain the policy distribution. The inference deployment module is used to generate multicast trees or forwarding paths that connect source nodes and multiple destination nodes step by step, using the same legal action mask generation rules as in the training phase.

15. The optimization system according to claim 14, characterized in that, The multicast request includes a source node and multiple destination nodes; the constraint processing of edge action scoring in the policy distribution generation module includes masking and normalization processing.

16. The system according to claim 14, characterized in that, The state construction module constructs a current multicast tree state matrix, a normalized bandwidth matrix, a normalized delay matrix, and a normalized packet loss rate matrix based on the network topology and the link state, and constructs a state input based on the current multicast tree state matrix, the normalized bandwidth matrix, the normalized delay matrix, and the normalized packet loss rate matrix.

17. The system according to claim 14, characterized in that, The model training module adopts a dual-Q network architecture and maintains the target Q network through soft updates. An adaptive temperature coefficient is introduced during training to adjust the weights of the policy entropy.

Images