Private domain live broadcast bandwidth resource allocation and routing method and system based on reinforcement learning

Abstract

The present invention provides a private domain live broadcast bandwidth resource allocation and routing method and system based on reinforcement learning, which relates to the field of computer network technology, including collecting real-time data such as the number of users in the live broadcast room, geographical distribution, terminal type, content type, network delay, packet loss rate, bandwidth utilization rate, etc., and inputting the data into a pre-trained reinforcement learning model; the model is based on state space, uses a deep neural network to extract features and combines bandwidth resource constraints, and adopts a policy gradient algorithm to generate bandwidth allocation and routing strategies; according to the strategy, a software-defined network controller adjusts bandwidth allocation parameters and routing table configuration in real time to achieve differentiated bandwidth allocation and path optimization, and establishes a nearby access edge node based on the user's geographical distribution, and dynamically adjusts the path when congested.

Description

Technical Field

[0001] The present invention relates to computer network technology, and in particular to a private domain live broadcast bandwidth resource allocation and routing method and system based on reinforcement learning. Background Art

[0002] The bandwidth allocation method for traditional private domain live broadcasts usually adopts static allocation or rule-based dynamic allocation. The static allocation method cannot be adjusted according to the real-time situation of the live broadcast room, which easily leads to waste or shortage of bandwidth resources. The rule-based dynamic allocation method relies on pre-set rules and is difficult to adapt to complex network environments and changing user needs, often resulting in unreasonable bandwidth allocation and affecting the user viewing experience. In addition, traditional routing strategies are mostly statically configured, lacking flexibility, and unable to dynamically adjust the data transmission path according to network conditions, which easily causes network congestion and delays.

[0003] Lack of flexibility in bandwidth allocation: Traditional bandwidth allocation methods are difficult to adapt to dynamic changes such as the concurrency of multiple live broadcast rooms and fluctuations in user scale in private domain live broadcast scenarios, resulting in low bandwidth resource utilization and poor user experience quality in some live broadcast rooms.

[0004] The routing strategy is not smart enough: Static routing or simple dynamic routing strategies cannot be optimized according to the real-time network conditions, which can easily lead to poor data transmission paths, increase network latency and packet loss rate, and affect the quality of live broadcasts.

[0005] Lack of refined control over user experience quality: Traditional bandwidth allocation and routing methods lack effective evaluation and control of user experience quality, making it difficult to provide differentiated services based on the needs of different users and network conditions, resulting in low user satisfaction. Summary of the invention

[0006] The embodiments of the present invention provide a private domain live broadcast bandwidth resource allocation and routing method and system based on reinforcement learning, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a private domain live broadcast bandwidth resource allocation and routing method based on reinforcement learning, including:

[0009] Collecting real-time data information of multiple live broadcast rooms in a private domain live broadcast network, the real-time data information includes the number of live broadcast room users, user geographical distribution, user terminal type, live broadcast content type, network delay, packet loss rate, and bandwidth utilization; inputting the real-time data information into a pre-trained reinforcement learning model, the state space of the reinforcement learning model includes bandwidth resource allocation status, network topology status, and user experience quality status;

[0010] The real-time data information is processed based on the reinforcement learning model, features are extracted through a deep neural network, and a bandwidth allocation strategy and a routing strategy are generated by a policy gradient algorithm in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution; sampling based on the bandwidth allocation probability distribution and the routing decision probability distribution to obtain a decision action, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality indicator and a system resource utilization rate; calculating a policy gradient according to the reward value, and updating the parameters of the policy network through back propagation;

[0011] According to the bandwidth allocation strategy and routing strategy, the bandwidth allocation parameters and routing table configuration in the network are adjusted in real time through the software-defined network controller, differentiated bandwidth allocation is performed for each live broadcast room and the data transmission path is optimized. The software-defined network controller establishes edge nodes with nearby access based on the user's geographical distribution, and dynamically adjusts the data transmission path when the network is congested.

[0012] In an optional embodiment,

[0013] The real-time data information is processed based on the reinforcement learning model, features are extracted through a deep neural network, and a bandwidth allocation strategy and a routing strategy are generated by using a policy gradient algorithm in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution. The steps include:

[0014] The state vectors of the bandwidth resource allocation state, network topology state and user experience quality state in the live broadcast network are input into the feature importance evaluation layer, and the state vectors are converted into state features through feature engineering. The feature importance evaluation layer constructs a correlation matrix between state features and network performance indicators based on historical decision data, analyzes the influence of each state feature on network performance based on the correlation matrix to obtain feature importance weights, and optimizes the feature importance weights according to real-time decision feedback information. The input state features are adaptively selected based on the optimized feature importance weights, and the selected state features are input into the input layer of the multi-layer perceptron. The hidden layer of the multi-layer perceptron compresses the state features through the ReLU activation function and obtains a low-dimensional compressed feature vector through residual connection;

[0015] The low-dimensional compressed feature vector is input into the spatiotemporal attention module, the spatial attention submodule of the spatiotemporal attention module obtains the spatial dimension attention weight by calculating the similarity of the state features of different network nodes and links, and the temporal attention submodule of the spatiotemporal attention module obtains the temporal dimension attention weight by calculating the correlation between the feature vectors of different time steps and the current state; the spatial dimension attention weight and the temporal dimension attention weight are respectively weightedly fused with the corresponding features to obtain an enhanced feature representation, and the enhanced feature representation is input into the long short-term memory network unit, the long short-term memory network unit comprises a forget gate, an input gate and an output gate, wherein the forget gate is used to control the degree of forgetting of historical state information, the input gate is used to control the degree of input of current state information, and the output gate is used to control the degree of output of the memory unit state; the long short-term memory network unit extracts the temporal features of the low-dimensional compressed feature vector based on the gate structure and outputs the hidden state;

[0016] The hidden state is input into a two-layer strategy network, which includes a bandwidth allocation layer and a routing decision layer, wherein the bandwidth allocation layer generates a bandwidth allocation probability distribution through a fully connected structure and a Softmax function, and the routing decision layer generates a routing decision probability distribution through a fully connected structure and a Softmax function.

[0017] In an optional embodiment,

[0018] The steps of training the reinforcement learning model include:

[0019] Construct an experience replay pool to store training samples. The training samples are triples including a state vector, an action vector, and a reward value. The state vector includes the bandwidth resource allocation state, the network topology state, and the user experience quality state. The action vector includes the bandwidth allocation decision and the routing decision. The reward value is calculated by the user experience quality index and the system resource utilization index.

[0020] Collect historical decision data to pre-process the state vector to obtain state features, construct a correlation matrix between the state features and network performance indicators, and obtain feature importance weights by performing singular value decomposition on the correlation matrix; construct a multi-layer perceptron autoencoder with residual connections according to the feature importance weights, compress the state vector into a low-dimensional compressed feature vector and reconstruct it, and train the multi-layer perceptron autoencoder by minimizing the weighted sum of the reconstruction error and the sparse regularization term;

[0021] Randomly sampling a batch of training data from the experience replay pool, calculating the log-likelihood probability of the current strategy based on the low-dimensional compressed feature vector and the action vector, multiplying the log-likelihood probability by the reward value to obtain the strategy objective function, and updating the strategy network parameters by maximizing the expected value of the strategy objective function; setting the target network, calculating the parameter update amount of the target network by using momentum accumulation, and adaptively adjusting the parameter update step size based on the current training loss;

[0022] The strategy entropy value is calculated according to the strategy distribution, the exploration probability is adaptively adjusted based on the strategy entropy value, and the reward value is smoothed by short-term and long-term double-scale exponential moving average; the state importance score is calculated based on the feature importance weight, the linear fusion weight of the user experience quality index and the system resource utilization index is dynamically adjusted according to the state importance score, the linear fusion weight is projected into the preset weight range for normalization, and the smoothed reward value is stored again in the experience replay pool.

[0023] In an optional embodiment,

[0024] The steps of obtaining a decision action based on the bandwidth allocation probability distribution and the routing decision probability distribution sampling, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality index and a system resource utilization rate include:

[0025] Importance sampling is performed on the bandwidth allocation probability distribution to obtain a candidate bandwidth allocation set, and routing decision sampling is performed on each bandwidth allocation scheme in the candidate bandwidth allocation set in combination with the routing decision probability distribution using the Gumbel-Softmax reparameterization technology to obtain a routing decision matching each bandwidth allocation scheme, thereby forming a decision action pair set;

[0026] For each group of decision actions in the decision action pair set, an end-to-end delay evaluation model is constructed to calculate the transmission delay, a video quality evaluation function is established to evaluate the impact of bandwidth allocation on video bit rate and clarity, and a packet loss rate detection function is used to evaluate network transmission reliability. The evaluation results of transmission delay, video quality, and network transmission reliability are normalized and weightedly integrated to obtain a user experience quality indicator;

[0027] Based on each group of decision actions in the decision action pair set, link bandwidth utilization, routing path utilization and load balancing degree are calculated, and the link bandwidth utilization, routing path utilization and load balancing degree are normalized and weightedly integrated to obtain system resource utilization;

[0028] Normalizing the user experience quality index and system resource utilization, calculating the fluctuation degree of the user experience quality index and the system resource utilization based on the historical decision effect, dynamically adjusting the adaptive fusion weight coefficient according to the fluctuation degree, and weighting the normalized user experience quality index and system resource utilization according to the adaptive fusion weight coefficient to obtain the original reward value;

[0029] The original reward value is subjected to exponential moving average to obtain a smoothed reward value, a final decision action is selected based on the smoothed reward value using a decreasing exploration probability strategy, and the final decision action and its corresponding smoothed reward value are stored in an experience replay pool for policy network training.

[0030] In an optional embodiment,

[0031] Importance sampling is performed on the bandwidth allocation probability distribution to obtain a candidate bandwidth allocation set, and routing decision sampling is performed on each bandwidth allocation scheme in the candidate bandwidth allocation set in combination with the routing decision probability distribution using the Gumbel-Softmax reparameterization technology to obtain a routing decision matching each bandwidth allocation scheme. The steps of forming a decision action pair set include:

[0032] Adaptive importance sampling is performed on the bandwidth allocation probability distribution, wherein the adaptive importance sampling includes calculating the importance weight of the bandwidth allocation scheme, wherein the importance weight is composed of the current state quality score and the historical decision effect score, wherein the historical decision effect score is calculated based on the historical success rate and the volatility of the decision value, and the weight coefficients of the historical success rate and the volatility of the decision value are dynamically adjusted according to the system stability requirements;

[0033] Adopting a hierarchical attention mechanism to optimize and screen the bandwidth allocation schemes of adaptive importance sampling, calculating the attention weight of the bandwidth resource dimension, reordering the bandwidth allocation schemes based on the attention weight of the bandwidth resource dimension, and selecting multiple bandwidth allocation schemes with the highest scores to form a candidate bandwidth allocation set;

[0034] Based on each bandwidth allocation scheme in the candidate bandwidth allocation set, the routing decision space is divided into multiple subspaces, the Gumbel-Softmax reparameterization technology is used in each subspace to sample the routing decision, the temperature parameter of each subspace is updated through adaptive learning, and the sampling results of each subspace are merged to obtain the routing decision corresponding to each bandwidth allocation scheme;

[0035] Perform dynamic quality constraint checks on decision pairs consisting of bandwidth allocation plans and routing decisions, calculate the matching degree of bandwidth and routing and the degree to which the decision pairs satisfy network constraints, dynamically adjust the quality constraint threshold based on the historical decision quality change trend, and retain the decision pairs that meet the quality constraint threshold requirements to form the final decision action pair set.

[0036] In an optional embodiment,

[0037] According to the bandwidth allocation strategy and routing strategy, the bandwidth allocation parameters and routing table configuration in the network are adjusted in real time through the software-defined network controller, and differentiated bandwidth allocation is performed on each live broadcast room and the data transmission path is optimized. The software-defined network controller establishes the nearest access edge node based on the user's geographical distribution, and dynamically adjusts the data transmission path when the network is congested. The steps include:

[0038] Deploy a hierarchical software-defined network controller architecture in the network, wherein the hierarchical software-defined network controller architecture includes a central controller and multiple regional controllers, wherein the central controller and the regional controllers perform state synchronization via an east-west interface, and a consistent hashing algorithm is used between the regional controllers for load balancing;

[0039] Based on the user's geographical distribution information, a clustering algorithm is used to determine the deployment location of the edge node, and the computing resource configuration of the edge node is dynamically adjusted according to the user density; according to the bandwidth allocation strategy output by the reinforcement learning model, the bandwidth allocation quota of each live broadcast room is calculated, and the bandwidth limit field of the flow table is configured based on the bandwidth allocation strategy, which is sent to the network device through the controller, and the bandwidth allocation parameters are adjusted through the exponential weighted moving average algorithm; a weighted directed graph is constructed based on the routing strategy output by the reinforcement learning model, and the node weight of the weighted directed graph is calculated according to the processor load, memory usage and queue length, and the link weight of the weighted directed graph is calculated according to the bandwidth utilization, delay and packet loss rate;

[0040] Based on the weighted directed graph, the shortest path algorithm considering node weights and link weights is used to calculate the optimal routing path; based on the weighted directed graph, the software-defined network controller monitors the network link status in real time to obtain a congestion index, which is calculated based on queue length, delay and packet loss rate. When the congestion index exceeds a preset threshold, a set of alternative paths is calculated, and the optimal alternative path is selected after evaluating the performance indicators of each alternative path;

[0041] The path migration is performed in the way of establishing first and then switching. After the new path is established and the corresponding flow table is issued, the traffic is gradually migrated. After the traffic is completely migrated, the original path flow table is deleted. The survival status of the regional controller is detected through heartbeat messages. When the failure of the regional controller is detected, the election algorithm is triggered to select a new main controller, restore the control plane status and rebuild the network view, and recalculate and issue the flow table configuration corresponding to the bandwidth allocation and routing strategy.

[0042] In an optional embodiment,

[0043] The steps of calculating the optimal routing path using the shortest path algorithm considering node weights and link weights include:

[0044] The node weight and the link weight are respectively normalized and the comprehensive weight is calculated to obtain the node cost and the link cost; a path cost calculation model is constructed based on the node cost and the link cost, and the path cost is the sum of all node costs and link costs on the path;

[0045] The improved shortest path algorithm is used to calculate the routing path, including: when performing the path relaxation operation, the cumulative cost from the target node to the source node, the current node cost and the current link cost are weighted and combined based on the dynamic programming principle; a relaxation counter is set to record the number of relaxation times of each node, and when the number of relaxation times of a node exceeds the preset number, the node is temporarily skipped by using the backoff mechanism; a suboptimal path set is maintained during the path search process, and when the optimal path does not meet the constraint conditions, an alternative path is selected from the suboptimal path set;

[0046] Calculate the relative cost difference between the new and old paths, and trigger a path update operation when the relative cost difference exceeds a preset threshold; establish a new routing path based on the path update operation and issue a corresponding flow table configuration, and evaluate the end-to-end delay, load balancing and path stability of the routing path; dynamically update the comprehensive weight according to the evaluation result, and use the updated comprehensive weight for subsequent path cost calculations.

[0047] According to a second aspect of the embodiments of the present invention,

[0048] Provides a private domain live broadcast bandwidth resource allocation and routing system based on reinforcement learning, including:

[0049] The first unit is used to collect real-time data information of multiple live broadcast rooms in the private domain live broadcast network, wherein the real-time data information includes the number of users in the live broadcast room, the geographical distribution of users, the type of user terminals, the type of live broadcast content, network delay, packet loss rate, and bandwidth utilization rate; the real-time data information is input into a pre-trained reinforcement learning model, wherein the state space of the reinforcement learning model includes bandwidth resource allocation state, network topology state, and user experience quality state;

[0050] The second unit is used to process the real-time data information based on the reinforcement learning model, extract features through a deep neural network, and use a policy gradient algorithm to generate a bandwidth allocation strategy and a routing strategy in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution; sampling based on the bandwidth allocation probability distribution and the routing decision probability distribution to obtain a decision action, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality indicator and a system resource utilization rate; calculating a policy gradient according to the reward value, and updating the parameters of the policy network through back propagation;

[0051] The third unit is used to adjust the bandwidth allocation parameters and routing table configuration in the network in real time through the software-defined network controller according to the bandwidth allocation strategy and routing strategy, perform differentiated bandwidth allocation for each live broadcast room and optimize the data transmission path. The software-defined network controller establishes edge nodes with nearby access based on the geographical distribution of users, and dynamically adjusts the data transmission path when the network is congested.

[0052] According to a third aspect of the embodiments of the present invention,

[0053] An electronic device is provided, comprising:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] A fourth aspect of the embodiments of the present invention is:

[0058] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0059] The present invention uses a reinforcement learning model to analyze information such as the number of users, geographical distribution, terminal type, live broadcast content, etc. in real time, and dynamically adjusts bandwidth allocation and routing strategies in combination with network conditions such as network delay and packet loss rate, thereby ensuring the smoothness of live broadcast, reducing freezes and delays, and improving user viewing experience.

[0060] The bandwidth allocation strategy generated by the policy gradient algorithm in the present invention can differentially allocate bandwidth resources according to the real-time needs of different live broadcast rooms, avoid resource waste, improve the overall bandwidth utilization efficiency, and support the concurrent operation of more live broadcast rooms under limited resource conditions.

[0061] The present invention utilizes a software-defined network controller for real-time control, can achieve nearby access based on the user's geographic location, and dynamically adjust the data transmission path when the network is congested, thereby improving the flexibility and adaptability of the network and ensuring the stability and reliability of the live broadcast service. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a process of a private domain live broadcast bandwidth resource allocation and routing method based on reinforcement learning according to an embodiment of the present invention;

[0063] Figure 2 This is a structural diagram of a private domain live broadcast bandwidth resource allocation and routing system based on reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of a method for allocating and routing bandwidth resources for private domain live broadcasting based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S1. Collect real-time data information of multiple live broadcast rooms in the private domain live broadcast network, wherein the real-time data information includes the number of users in the live broadcast room, the geographical distribution of users, the type of user terminals, the type of live broadcast content, network delay, packet loss rate, and bandwidth utilization rate; input the real-time data information into a pre-trained reinforcement learning model, wherein the state space of the reinforcement learning model includes bandwidth resource allocation status, network topology status, and user experience quality status;

[0068] S2. Process the real-time data information based on the reinforcement learning model, extract features through a deep neural network, combine bandwidth resource constraints, and use a policy gradient algorithm to generate a bandwidth allocation strategy and a routing strategy, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution; sampling based on the bandwidth allocation probability distribution and the routing decision probability distribution to obtain a decision action, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality indicator and a system resource utilization rate; calculating a policy gradient according to the reward value, and updating the parameters of the policy network through back propagation;

[0069] S3. According to the bandwidth allocation strategy and routing strategy, the bandwidth allocation parameters and routing table configuration in the network are adjusted in real time through the software-defined network controller, differentiated bandwidth allocation is performed for each live broadcast room and the data transmission path is optimized. The software-defined network controller establishes edge nodes with nearby access based on the user's geographical distribution, and dynamically adjusts the data transmission path when the network is congested.

[0070] In an optional embodiment,

[0071] The real-time data information is processed based on the reinforcement learning model, features are extracted through a deep neural network, and a bandwidth allocation strategy and a routing strategy are generated by using a policy gradient algorithm in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution. The steps include:

[0072] The state vectors of the bandwidth resource allocation state, network topology state and user experience quality state in the live broadcast network are input into the feature importance evaluation layer, and the state vectors are converted into state features through feature engineering. The feature importance evaluation layer constructs a correlation matrix between state features and network performance indicators based on historical decision data, analyzes the influence of each state feature on network performance based on the correlation matrix to obtain feature importance weights, and optimizes the feature importance weights according to real-time decision feedback information. The input state features are adaptively selected based on the optimized feature importance weights, and the selected state features are input into the input layer of the multi-layer perceptron. The hidden layer of the multi-layer perceptron compresses the state features through the ReLU activation function and obtains a low-dimensional compressed feature vector through residual connection;

[0073] The low-dimensional compressed feature vector is input into the spatiotemporal attention module, the spatial attention submodule of the spatiotemporal attention module obtains the spatial dimension attention weight by calculating the similarity of the state features of different network nodes and links, and the temporal attention submodule of the spatiotemporal attention module obtains the temporal dimension attention weight by calculating the correlation between the feature vectors of different time steps and the current state; the spatial dimension attention weight and the temporal dimension attention weight are respectively weightedly fused with the corresponding features to obtain an enhanced feature representation, and the enhanced feature representation is input into the long short-term memory network unit, the long short-term memory network unit comprises a forget gate, an input gate and an output gate, wherein the forget gate is used to control the degree of forgetting of historical state information, the input gate is used to control the degree of input of current state information, and the output gate is used to control the degree of output of the memory unit state; the long short-term memory network unit extracts the temporal features of the low-dimensional compressed feature vector based on the gate structure and outputs the hidden state;

[0074] The hidden state is input into a two-layer strategy network, which includes a bandwidth allocation layer and a routing decision layer, wherein the bandwidth allocation layer generates a bandwidth allocation probability distribution through a fully connected structure and a Softmax function, and the routing decision layer generates a routing decision probability distribution through a fully connected structure and a Softmax function.

[0075] For example, first, real-time data information of the live broadcast network is collected, including bandwidth resource allocation status, network topology status, and user experience quality status. For example, bandwidth resource allocation status can be expressed as the available bandwidth of each link, network topology status can be expressed as the connection relationship between network nodes, and user experience quality status can be expressed as indicators such as user delay, packet loss rate, and video clarity. These status information are integrated into a state vector.

[0076] Next, the collected state vectors are subjected to feature extraction and importance evaluation. The state vectors are input into the feature importance evaluation layer. This layer builds a correlation matrix based on the relationship between historical decision data and network performance indicators. For example, if past decision data shows that increasing the bandwidth allocation of a link will significantly reduce user latency, then the bandwidth allocation state feature of this link will be given a higher importance weight. At the same time, the system dynamically adjusts the feature importance weights based on real-time decision feedback information. For example, if the system detects that the quality of experience of a user has degraded, the weight of the state feature associated with the user will be increased. Based on the optimized weights, important state features are adaptively selected. Then, the selected state features are input into the multi-layer perceptron. The hidden layer of the multi-layer perceptron compresses the state features using the ReLU activation function and obtains a low-dimensional compressed feature vector through residual connection. For example, a state feature vector containing 100 elements may become a low-dimensional vector containing only 20 elements after being compressed by the multi-layer perceptron.

[0077] Then, the low-dimensional compressed feature vector is input into the spatiotemporal attention module. The spatiotemporal attention module contains the spatial attention submodule and the temporal attention submodule. The spatial attention submodule calculates the similarity of the state features of different network nodes and links to obtain the spatial dimension attention weight. For example, if the state features of two nodes are very similar, they will be assigned a higher spatial attention weight. The temporal attention submodule calculates the correlation between the feature vectors of different time steps and the current state to obtain the temporal dimension attention weight. For example, if the current state is very similar to the state of the previous time step, the feature vector of the previous time step will be assigned a higher attention weight. The spatial dimension attention weight and the temporal dimension attention weight are weightedly fused with the corresponding features to obtain the enhanced feature representation.

[0078] Next, the enhanced feature representation is input into the long short-term memory network unit (LSTM). The LSTM unit contains a forget gate, an input gate, and an output gate. The forget gate controls the degree of forgetting of historical state information, the input gate controls the degree of input of current state information, and the output gate controls the degree of output of the memory unit state. The LSTM unit extracts time series features from low-dimensional compressed feature vectors based on the gated structure and outputs the hidden state.

[0079] Finally, the hidden state output by the LSTM unit is input into the two-layer policy network. The two-layer policy network includes a bandwidth allocation layer and a routing decision layer. The bandwidth allocation layer generates a bandwidth allocation probability distribution through a fully connected structure and a Softmax function. For example, the bandwidth allocation layer may output a probability distribution indicating that the available bandwidth is allocated to different links in different proportions. The routing decision layer generates a routing decision probability distribution through a fully connected structure and a Softmax function. For example, the routing decision layer may output a probability distribution indicating the probability of selecting different routes. Based on these two probability distributions, the system selects a bandwidth allocation strategy and a routing strategy. Then, the system applies the selected strategy to the live broadcast network and observes the network performance and user experience quality for updating the reinforcement learning model.

[0080] The present invention can effectively reduce user delay and packet loss rate, improve video clarity, and thus enhance user experience quality by dynamically adjusting bandwidth allocation and route selection. It can maximize the use of available bandwidth under bandwidth resource constraints, avoid bandwidth waste, and improve bandwidth resource utilization. It can dynamically adjust strategies according to real-time network status and user experience quality, thereby enhancing the network's adaptability to dynamically changing environments.

[0081] In an optional embodiment,

[0082] The steps of training the reinforcement learning model include:

[0083] Construct an experience replay pool to store training samples. The training samples are triples including a state vector, an action vector, and a reward value. The state vector includes the bandwidth resource allocation state, the network topology state, and the user experience quality state. The action vector includes the bandwidth allocation decision and the routing decision. The reward value is calculated by the user experience quality index and the system resource utilization index.

[0084] Collect historical decision data to pre-process the state vector to obtain state features, construct a correlation matrix between the state features and network performance indicators, and obtain feature importance weights by performing singular value decomposition on the correlation matrix; construct a multi-layer perceptron autoencoder with residual connections according to the feature importance weights, compress the state vector into a low-dimensional compressed feature vector and reconstruct it, and train the multi-layer perceptron autoencoder by minimizing the weighted sum of the reconstruction error and the sparse regularization term;

[0085] Randomly sampling a batch of training data from the experience replay pool, calculating the log-likelihood probability of the current strategy based on the low-dimensional compressed feature vector and the action vector, multiplying the log-likelihood probability by the reward value to obtain the strategy objective function, and updating the strategy network parameters by maximizing the expected value of the strategy objective function; setting the target network, calculating the parameter update amount of the target network by using momentum accumulation, and adaptively adjusting the parameter update step size based on the current training loss;

[0086] The strategy entropy value is calculated according to the strategy distribution, the exploration probability is adaptively adjusted based on the strategy entropy value, and the reward value is smoothed by short-term and long-term double-scale exponential moving average; the state importance score is calculated based on the feature importance weight, the linear fusion weight of the user experience quality index and the system resource utilization index is dynamically adjusted according to the state importance score, the linear fusion weight is projected into the preset weight range for normalization, and the smoothed reward value is stored again in the experience replay pool.

[0087] Exemplarily, first, an experience replay pool is constructed. The experience replay pool is used to store training samples, each sample containing a state vector, an action vector, and a reward value. The state vector contains bandwidth resource allocation state, network topology state, and user experience quality state. For example, the bandwidth resource allocation state can be expressed as the proportion of available bandwidth for each link, the network topology state can be expressed as a node connection relationship matrix, and the user experience quality state can be expressed as indicators such as average delay and packet loss rate. The action vector contains bandwidth allocation decisions and routing decisions. For example, the bandwidth allocation decision can be expressed as the proportion of bandwidth allocated to each user, and the routing decision can be expressed as the forwarding path for each data packet. The reward value is calculated by the user experience quality indicator and the system resource utilization indicator. For example, the reward value can be the weighted sum of the user experience quality indicator and the system resource utilization indicator.

[0088] Next, the state vector is preprocessed to extract key features. Historical decision data is collected to construct a correlation matrix between state features and network performance indicators. For example, the changes in average delay under different bandwidth allocation states are counted. The correlation matrix is ​​subjected to singular value decomposition to obtain feature importance weights. The larger the weight value, the greater the impact of the feature on network performance. A multilayer perceptron autoencoder with residual connections is constructed based on the feature importance weights. The autoencoder compresses the state vector into a low-dimensional compressed feature vector and reconstructs it. For example, a state vector containing 10 elements can be compressed into a low-dimensional vector of 3 elements. The autoencoder is trained by minimizing the weighted sum of the reconstruction error and the sparse regularization term. This allows the compressed feature vector to retain the most important information.

[0089] Then, randomly sample a batch of training data from the experience replay pool. Calculate the log-likelihood probability of the current policy based on the low-dimensional compressed feature vector and the action vector. The log-likelihood probability represents the probability of taking a specific action in the current state. Multiply the log-likelihood probability with the reward value to get the policy objective function. Update the policy network parameters by maximizing the expected value of the policy objective function. Set the target network and calculate the parameter update amount of the target network using momentum accumulation. For example, the update amount of the target network parameters is the weighted average of the difference between the current policy network parameters and the target network parameters. Adaptively adjust the parameter update step size based on the current training loss. The smaller the loss, the smaller the step size to avoid oscillation.

[0090] In addition, the policy entropy value is calculated based on the policy distribution. The policy entropy value indicates the randomness of the policy. The larger the entropy value, the more random the policy. The exploration probability is adaptively adjusted based on the policy entropy value. The smaller the entropy value, the greater the exploration probability, to encourage the exploration of new policies. The reward value is smoothed by short-term and long-term dual-scale exponential moving average. For example, the short-term smoothing window is 5 and the long-term smoothing window is 20. The state importance score is calculated based on the feature importance weight. The higher the state importance score, the greater the impact of the state on network performance. The linear fusion weights of the user experience quality index and the system resource utilization index are dynamically adjusted according to the state importance score. For example, if the user experience quality is poor in the current state, the weight of the user experience quality index is increased. The linear fusion weight is projected into the preset weight range for normalization to obtain the smoothed reward value, and it is stored back in the experience replay pool.

[0091] The present invention can effectively reduce network delay and packet loss rate by dynamically adjusting bandwidth resource allocation strategy and routing strategy, thereby improving user experience quality. By optimizing bandwidth resource allocation and routing strategy, resource waste can be avoided and system resource utilization can be improved. Through state feature extraction, state importance score calculation and reward value smoothing and other technologies, the model can be better adapted to different network environments and user needs.

[0092] In an optional embodiment,

[0093] The steps of obtaining a decision action based on the bandwidth allocation probability distribution and the routing decision probability distribution sampling, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality index and a system resource utilization rate include:

[0094] Importance sampling is performed on the bandwidth allocation probability distribution to obtain a candidate bandwidth allocation set, and routing decision sampling is performed on each bandwidth allocation scheme in the candidate bandwidth allocation set in combination with the routing decision probability distribution using the Gumbel-Softmax reparameterization technology to obtain a routing decision matching each bandwidth allocation scheme, thereby forming a decision action pair set;

[0095] For each group of decision actions in the decision action pair set, an end-to-end delay evaluation model is constructed to calculate the transmission delay, a video quality evaluation function is established to evaluate the impact of bandwidth allocation on video bit rate and clarity, and a packet loss rate detection function is used to evaluate network transmission reliability. The evaluation results of transmission delay, video quality, and network transmission reliability are normalized and weightedly integrated to obtain a user experience quality indicator;

[0096] Based on each group of decision actions in the decision action pair set, link bandwidth utilization, routing path utilization and load balancing degree are calculated, and the link bandwidth utilization, routing path utilization and load balancing degree are normalized and weightedly integrated to obtain system resource utilization;

[0097] Normalizing the user experience quality index and system resource utilization, calculating the fluctuation degree of the user experience quality index and the system resource utilization based on the historical decision effect, dynamically adjusting the adaptive fusion weight coefficient according to the fluctuation degree, and weighting the normalized user experience quality index and system resource utilization according to the adaptive fusion weight coefficient to obtain the original reward value;

[0098] The original reward value is subjected to exponential moving average to obtain a smoothed reward value, a final decision action is selected based on the smoothed reward value using a decreasing exploration probability strategy, and the final decision action and its corresponding smoothed reward value are stored in an experience replay pool for policy network training.

[0099] Exemplarily, first, the bandwidth allocation probability distribution is sampled to obtain a set of candidate bandwidth allocation schemes. For example, assuming that the current network allocatable bandwidth is 100 Mbps, the bandwidth allocation probability distribution is divided into multiple intervals, such as [0, 20 Mbps], [20 Mbps, 40 Mbps], [40 Mbps, 60 Mbps], [60 Mbps, 80 Mbps], and [80 Mbps, 100 Mbps]. Bandwidth allocation values ​​are selected from these intervals by importance sampling, such as sampling 15 Mbps from the first interval, sampling 30 Mbps from the second interval, and so on, and finally a candidate bandwidth allocation set is obtained, such as {15 Mbps, 30 Mbps, 50 Mbps, 70 Mbps, 90 Mbps}.

[0100] Next, for each candidate bandwidth allocation scheme, the routing decision probability distribution is combined to perform routing selection. The Gumbel-Softmax reparameterization technique is used to sample the routing decision probability distribution to obtain the routing decision corresponding to each bandwidth allocation scheme. For example, assuming there are three optional routing paths, for the 15Mbps bandwidth allocation scheme, the Gumbel-Softmax technique is used to calculate the probabilities of selecting the three paths to be 0.2, 0.3, and 0.5, respectively, and finally the routing path 3 is sampled. The bandwidth allocation scheme and the corresponding routing decision are combined to form a set of decision action pairs, such as {(15Mbps, path 3), (30Mbps, path 2), (50Mbps, path 1), (70Mbps, path 2), (90Mbps, path 3)}.

[0101] Then, each decision action pair is evaluated. An end-to-end delay evaluation model is constructed to calculate the delay of data packets transmitted in the network. A video quality evaluation function is established to evaluate the impact of bandwidth allocation on video bit rate and clarity. The reliability of network transmission is evaluated through the packet loss rate detection function. For example, for the decision action group (15Mbps, path 3), the calculated transmission delay is 50ms, the video quality score is 0.8, and the packet loss rate is 0.01. After normalizing the transmission delay, video quality, and network transmission reliability, for example, they are all mapped between 0 and 1, and then weighted fusion is performed according to the preset weights to obtain the user experience quality index. Assuming the weights are 0.4, 0.5, and 0.1 respectively, the user experience quality index of this decision action is 0.4×(50ms normalized value)+0.5×0.8+0.1×(0.01 normalized value).

[0102] At the same time, the system resource utilization corresponding to each group of decision actions is calculated. Link bandwidth utilization, routing path utilization and load balancing are calculated. After these indicators are normalized, for example, they are all mapped to between 0 and 1, and weighted fusion is performed according to the preset weights to obtain the system resource utilization.

[0103] Next, normalize the user experience quality index and system resource utilization. Calculate the fluctuation degree of the user experience quality index and the system resource utilization based on the historical decision effect. Dynamically adjust the adaptive fusion weight coefficient according to the fluctuation degree. Weight the normalized user experience quality index and system resource utilization according to the adaptive fusion weight coefficient to obtain the original reward value.

[0104] The original reward value is exponentially moved averaged to obtain a smoothed reward value. Based on the smoothed reward value, the final decision action is selected using a decreasing exploration probability strategy. For example, as the training progresses, the exploration probability gradually decreases, and the decision action with a higher smoothed reward value is more likely to be selected. The final decision action and its corresponding smoothed reward value are stored in the experience replay pool for subsequent policy network training.

[0105] The present invention can effectively reduce transmission delay, improve video clarity and network transmission reliability, thereby enhancing user experience by comprehensively considering the impact of bandwidth allocation and route selection on user experience quality. By considering system resource utilization indicators such as link bandwidth utilization, routing path utilization and load balancing, network congestion can be effectively avoided and resource utilization efficiency can be improved. By dynamically adjusting the fusion weight coefficient according to historical decision-making effects, the algorithm can adaptively learn and optimize strategies, thereby improving the accuracy and stability of decision-making.

[0106] In an optional embodiment,

[0107] Importance sampling is performed on the bandwidth allocation probability distribution to obtain a candidate bandwidth allocation set, and routing decision sampling is performed on each bandwidth allocation scheme in the candidate bandwidth allocation set in combination with the routing decision probability distribution using the Gumbel-Softmax reparameterization technology to obtain a routing decision matching each bandwidth allocation scheme. The steps of forming a decision action pair set include:

[0108] Adaptive importance sampling is performed on the bandwidth allocation probability distribution, wherein the adaptive importance sampling includes calculating the importance weight of the bandwidth allocation scheme, wherein the importance weight is composed of the current state quality score and the historical decision effect score, wherein the historical decision effect score is calculated based on the historical success rate and the volatility of the decision value, and the weight coefficients of the historical success rate and the volatility of the decision value are dynamically adjusted according to the system stability requirements;

[0109] Adopting a hierarchical attention mechanism to optimize and screen the bandwidth allocation schemes of adaptive importance sampling, calculating the attention weight of the bandwidth resource dimension, reordering the bandwidth allocation schemes based on the attention weight of the bandwidth resource dimension, and selecting multiple bandwidth allocation schemes with the highest scores to form a candidate bandwidth allocation set;

[0110] Based on each bandwidth allocation scheme in the candidate bandwidth allocation set, the routing decision space is divided into multiple subspaces, the Gumbel-Softmax reparameterization technology is used in each subspace to sample the routing decision, the temperature parameter of each subspace is updated through adaptive learning, and the sampling results of each subspace are merged to obtain the routing decision corresponding to each bandwidth allocation scheme;

[0111] Perform dynamic quality constraint checks on decision pairs consisting of bandwidth allocation plans and routing decisions, calculate the matching degree of bandwidth and routing and the degree to which the decision pairs satisfy network constraints, dynamically adjust the quality constraint threshold based on the historical decision quality change trend, and retain the decision pairs that meet the quality constraint threshold requirements to form the final decision action pair set.

[0112] Exemplarily, first, adaptive importance sampling is performed on the bandwidth allocation probability distribution. The goal of this step is to screen out more promising candidate solutions from a large number of bandwidth allocation solutions. Specifically, the system calculates the importance weight of each bandwidth allocation solution. This weight consists of two parts: the current state quality score and the historical decision effect score. The current state quality score reflects the applicability of the bandwidth allocation solution under the current network state. For example, factors such as the degree of network congestion and node load can be considered. The historical decision effect score is calculated based on the historical success rate and decision value volatility. The historical success rate refers to the proportion of success of the solution in past use. Decision value volatility measures the stability of the effect of the solution under different circumstances. In order to adapt to different network environments, the system dynamically adjusts the weight coefficients of the historical success rate and decision value volatility according to the stability requirements. For example, if the network environment changes drastically, more attention will be paid to decision value volatility to select a more robust solution. Assume that there are three bandwidth allocation schemes A, B, and C, whose current state quality scores are 0.8, 0.7, and 0.6, respectively, their historical success rates are 0.9, 0.8, and 0.7, and their decision value volatility is 0.1, 0.2, and 0.3, respectively. Assuming that the current network environment is relatively stable, the weight coefficient of the historical success rate is 0.7, and the weight coefficient of the decision value volatility is 0.3, then the importance weight of scheme A is 0.8×0.7+0.9×0.3=0.83.

[0113] Next, the hierarchical attention mechanism is used to optimize and screen the bandwidth allocation schemes that have undergone importance sampling. The system calculates the attention weight of the bandwidth resource dimension, which reflects the impact of different bandwidth resource dimensions on the overall performance. For example, the bandwidth resources of some key links may be more important than other links. Based on the attention weight of the bandwidth resource dimension, the system reorders the bandwidth allocation schemes and selects the highest-scoring schemes to form a candidate bandwidth allocation set. Assume that the bandwidth resource dimension weight calculated by the attention mechanism is [0.5, 0.3, 0.2], and the scores of schemes A, B, and C in the three dimensions are [0.9, 0.8, 0.7], [0.7, 0.9, 0.6], and [0.8, 0.7, 0.9], respectively. Then the final score of scheme A is 0.9×0.5+0.8×0.3+0.7×0.2=0.83. Similarly, the scores of schemes B and C can be calculated, and the scheme with the highest score is selected to enter the candidate set.

[0114] Then, for each bandwidth allocation scheme in the candidate bandwidth allocation set, the system will perform routing decision sampling. First, the routing decision space is divided into multiple subspaces for more refined search. In each subspace, the Gumbel-Softmax reparameterization technique is used to sample routing decisions. This technique can effectively sample from the discrete routing decision space and maintain the differentiability of the sampling, which is convenient for subsequent optimization. The system will adaptively learn and update the temperature parameters of each subspace to control the randomness of the sampling. Finally, the sampling results of each subspace are merged to obtain the routing decision corresponding to each bandwidth allocation scheme.

[0115] Finally, dynamic quality constraint checks are performed on the decision pairs consisting of bandwidth allocation schemes and routing decisions. The system calculates the matching degree of bandwidth and routing and the degree to which the decision pairs satisfy network constraints. For example, whether the bandwidth allocation scheme can support the selected routing and whether the routing satisfies the network's latency, reliability and other constraints. In order to adapt to the dynamic changes of the network, the system dynamically adjusts the quality constraint threshold based on the historical decision quality change trend. Only decision pairs that meet the quality constraint threshold requirements will be retained to form the final decision action pair set.

[0116] The present invention can more effectively utilize limited bandwidth resources and avoid resource waste through adaptive importance sampling and hierarchical attention mechanism. By considering historical decision effects and dynamically adjusting quality constraint thresholds, the robustness of decisions can be improved and the dynamic changes of the network can be adapted. By jointly optimizing bandwidth allocation and routing decisions, the performance requirements of the network can be better met.

[0117] In an optional embodiment,

[0118] According to the bandwidth allocation strategy and routing strategy, the bandwidth allocation parameters and routing table configuration in the network are adjusted in real time through the software-defined network controller, and differentiated bandwidth allocation is performed on each live broadcast room and the data transmission path is optimized. The software-defined network controller establishes the nearest access edge node based on the user's geographical distribution, and dynamically adjusts the data transmission path when the network is congested. The steps include:

[0119] Deploy a hierarchical software-defined network controller architecture in the network, wherein the hierarchical software-defined network controller architecture includes a central controller and multiple regional controllers, wherein the central controller and the regional controllers perform state synchronization via an east-west interface, and a consistent hashing algorithm is used between the regional controllers for load balancing;

[0120] Based on the user's geographical distribution information, a clustering algorithm is used to determine the deployment location of the edge node, and the computing resource configuration of the edge node is dynamically adjusted according to the user density; according to the bandwidth allocation strategy output by the reinforcement learning model, the bandwidth allocation quota of each live broadcast room is calculated, and the bandwidth limit field of the flow table is configured based on the bandwidth allocation strategy, which is sent to the network device through the controller, and the bandwidth allocation parameters are adjusted through the exponential weighted moving average algorithm; a weighted directed graph is constructed based on the routing strategy output by the reinforcement learning model, and the node weight of the weighted directed graph is calculated according to the processor load, memory usage and queue length, and the link weight of the weighted directed graph is calculated according to the bandwidth utilization, delay and packet loss rate;

[0121] Based on the weighted directed graph, the shortest path algorithm considering node weights and link weights is used to calculate the optimal routing path; based on the weighted directed graph, the software-defined network controller monitors the network link status in real time to obtain a congestion index, which is calculated based on queue length, delay and packet loss rate. When the congestion index exceeds a preset threshold, a set of alternative paths is calculated, and the optimal alternative path is selected after evaluating the performance indicators of each alternative path;

[0122] The path migration is performed in the way of establishing first and then switching. After the new path is established and the corresponding flow table is issued, the traffic is gradually migrated. After the traffic is completely migrated, the original path flow table is deleted. The survival status of the regional controller is detected through heartbeat messages. When the failure of the regional controller is detected, the election algorithm is triggered to select a new main controller, restore the control plane status and rebuild the network view, and recalculate and issue the flow table configuration corresponding to the bandwidth allocation and routing strategy.

[0123] Exemplarily, first, a hierarchical software-defined network controller architecture is deployed. The architecture includes a central controller and multiple regional controllers. The central controller is responsible for global network policy formulation and resource coordination, and the regional controllers are responsible for managing network devices and resources in their respective areas. The central controller and the regional controllers synchronize states through the east-west interface, for example, using the OpenFlow protocol or the NETCONF protocol to ensure that all controllers have a global network view. A consistent hashing algorithm is used for load balancing between regional controllers. For example, the identifier of each live broadcast room is hashed, and then the live broadcast room is assigned to different regional controllers according to the hash value, thereby avoiding overloading of a single regional controller.

[0124] Next, edge nodes are deployed based on user geographic distribution information. Clustering algorithms, such as the K-Means algorithm, are used to cluster user geographic location information to determine the best deployment location for edge nodes. For example, the user's geographic location coordinates are used as input data, the number of cluster centers is set to the preset number of edge nodes, and the K-Means algorithm is used to iteratively calculate and finally determine the deployment location of each edge node. The computing resource configuration of edge nodes is dynamically adjusted according to user density. For example, the number of users within the coverage area of ​​each edge node is counted, and the CPU, memory, and bandwidth resources of the edge node are dynamically adjusted according to the number of users.

[0125] Then, according to the bandwidth allocation strategy output by the reinforcement learning model, the bandwidth allocation quota of each live broadcast room is calculated. The input of the reinforcement learning model may include information such as the number of users in the live broadcast room, the live broadcast quality requirements, and the network load status. The output of the model is the bandwidth allocation ratio of each live broadcast room. For example, a live broadcast room has 10,000 viewers, the live broadcast quality requires high definition, and the network load is low. The bandwidth allocation ratio output by the reinforcement learning model is 10%, then the bandwidth allocation quota of the live broadcast room is 10% of the current available bandwidth. Configure the bandwidth limit field of the flow table based on the bandwidth allocation strategy and send it to the network device through the controller. For example, use the OpenFlow protocol to send the flow table, and set the bandwidth limit field of the flow table to the calculated bandwidth allocation quota. And adjust the bandwidth allocation parameters through the exponentially weighted moving average algorithm. For example, according to the real-time network load, dynamically adjust the weight parameters of the exponentially weighted moving average algorithm to make bandwidth allocation more flexible.

[0126] Subsequently, a weighted directed graph is constructed based on the routing strategy output by the reinforcement learning model. The node weight is calculated based on the processor load, memory usage, and queue length. For example, if the node's processor load is 80%, the memory usage is 70%, and the queue length is 100 packets, the node weight can be set to 0.8×0.7×100=56. The link weight is calculated based on the bandwidth utilization, latency, and packet loss rate. For example, if the link's bandwidth utilization is 90%, the latency is 50ms, and the packet loss rate is 1%, the link weight can be set to 0.9×50×0.01=0.45.

[0127] Based on the weighted directed graph, the optimal routing path is calculated by using the shortest path algorithm that considers node weights and link weights, such as the Dijkstra algorithm.

[0128] Monitor the network link status in real time to obtain the congestion index. The congestion index is calculated based on the queue length, delay, and packet loss rate. For example, if the queue length exceeds 1,000 packets, the delay exceeds 100ms, and the packet loss rate exceeds 5%, the network is considered to be congested. When the congestion index exceeds the preset threshold, the set of alternative paths is calculated, and the performance indicators of each alternative path, such as bandwidth, delay, and packet loss rate, are evaluated to select the optimal alternative path.

[0129] The path migration is performed by establishing first and then switching. First, a new path is established and the corresponding flow table is issued, and then the traffic is gradually migrated. After the traffic is completely migrated, the original path flow table is deleted.

[0130] Finally, the survival status of the regional controller is detected through heartbeat messages. When the regional controller fails, the election algorithm is triggered to select a new master controller, restore the control plane state and rebuild the network view, and recalculate and send the flow table configuration corresponding to the bandwidth allocation and routing strategy. For example, the Raft algorithm is used to elect the master controller.

[0131] The present invention reduces the delay of users accessing live broadcast services by accessing the nearest edge node and dynamically allocating bandwidth, ensures the fluency of live broadcast, and improves the user's viewing experience. By dynamically adjusting bandwidth allocation and routing strategies through reinforcement learning models, resources can be dynamically allocated according to network load conditions to avoid resource waste and improve network resource utilization. The hierarchical controller architecture and the automatic switching mechanism after controller failure enhance the network's fault tolerance and ensure the stable operation of the network.

[0132] In an optional embodiment,

[0133] The steps of calculating the optimal routing path using the shortest path algorithm considering node weights and link weights include:

[0134] The node weight and the link weight are respectively normalized and the comprehensive weight is calculated to obtain the node cost and the link cost; a path cost calculation model is constructed based on the node cost and the link cost, and the path cost is the sum of all node costs and link costs on the path;

[0135] The improved shortest path algorithm is used to calculate the routing path, including: when performing the path relaxation operation, the cumulative cost from the target node to the source node, the current node cost and the current link cost are weighted and combined based on the dynamic programming principle; a relaxation counter is set to record the number of relaxation times of each node, and when the number of relaxation times of a node exceeds the preset number, the node is temporarily skipped by using the backoff mechanism; a suboptimal path set is maintained during the path search process, and when the optimal path does not meet the constraint conditions, an alternative path is selected from the suboptimal path set;

[0136] Calculate the relative cost difference between the new and old paths, and trigger a path update operation when the relative cost difference exceeds a preset threshold; establish a new routing path based on the path update operation and issue a corresponding flow table configuration, and evaluate the end-to-end delay, load balancing and path stability of the routing path; dynamically update the comprehensive weight according to the evaluation result, and use the updated comprehensive weight for subsequent path cost calculations.

[0137] Exemplarily, first, the node weights and link weights are normalized. For example, suppose there are three nodes A, B, and C in the network, and the node weights are 10, 20, and 30 respectively, and the weights of links AB, BC, and AC are 5, 15, and 10 respectively. The node weights are normalized to between 0 and 1, and the normalized weight of node A is 10 / (10+20+30)=0.17, node B is 0.33, and node C is 0.5. Similarly, the link weights are also normalized to between 0 and 1, and the normalized weight of link AB is 5 / (5+15+10)=0.17, link BC is 0.5, and link AC is 0.33.

[0138] Next, calculate the comprehensive weights of nodes and links. Here, a simple weighted average method is used. For example, the node weight accounts for 0.4 and the link weight accounts for 0.6. Then the cost of node A is 0.17×0.4=0.068, node B is 0.33×0.4=0.132, and node C is 0.5×0.4=0.2. The cost of link AB is 0.17×0.6=0.102, link BC is 0.5×0.6=0.3, and link AC is 0.33×0.6=0.198.

[0139] Then, a path cost calculation model is constructed. The cost of a path is equal to the sum of all node costs and link costs on the path. For example, the cost of path ABC is node A cost + link AB cost + node B cost + link BC cost + node C cost = 0.068 + 0.102 + 0.132 + 0.3 + 0.2 = 0.702.

[0140] The improved shortest path algorithm is used to calculate the routing path. In the path relaxation operation, based on the principle of dynamic programming, the cumulative cost from the target node to the source node, the current node cost, and the current link cost are weighted and combined. For example, assuming that the current node is B, the target node is C, the source node is A, and the cumulative cost from A to B is 0.17, then the path cost from B to C is 0.17+0.132+0.3=0.602.

[0141] Set a relaxation counter to record the number of relaxations of each node. When the number of relaxations of a node exceeds the preset number (for example, 10 times), a backoff mechanism is used to temporarily skip the node to avoid falling into a local optimal solution.

[0142] A set of suboptimal paths is maintained during the path search process. When the optimal path does not meet the constraints (e.g., insufficient bandwidth), an alternative path is selected from the set of suboptimal paths.

[0143] Calculate the relative cost difference between the new and old paths. For example, if the new path cost is 0.6, the old path cost is 0.7, and the preset threshold is 0.05, then the relative cost difference is (0.7-0.6) / 0.7=0.14, which exceeds the threshold.

[0144] When the relative cost difference exceeds the preset threshold, the path update operation is triggered. Based on the path update operation, a new routing path is established and the corresponding flow table configuration is issued.

[0145] Evaluate the end-to-end delay, load balancing, and path stability of the routing path. For example, obtain the end-to-end delay through network measurement tools, evaluate the load balancing based on link utilization, and evaluate the path stability based on the path switching frequency.

[0146] The comprehensive weight is dynamically updated based on the evaluation results. For example, if the path stability is poor, the proportion of node weight is increased and the proportion of link weight is reduced. The updated comprehensive weight is used for subsequent path cost calculation to form a closed-loop control.

[0147] The present invention can more accurately evaluate the path cost by considering the comprehensive weights of nodes and links, thereby finding a more optimal routing path and improving the utilization rate of network resources. By setting a relaxation counter and a backoff mechanism, the algorithm is prevented from falling into a local optimal solution. By maintaining a suboptimal path set, an alternative solution is provided when the optimal path is unavailable, thereby enhancing the stability of routing. By dynamically updating the comprehensive weight, the routing strategy is adaptively adjusted according to the network status, thereby improving the flexibility of routing and enabling it to better adapt to the dynamically changing network environment.

[0148] Figure 2 FIG. 1 is a structural diagram of a private domain live broadcast bandwidth resource allocation and routing system based on reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0149] The first unit is used to collect real-time data information of multiple live broadcast rooms in the private domain live broadcast network, wherein the real-time data information includes the number of users in the live broadcast room, the geographical distribution of users, the type of user terminals, the type of live broadcast content, network delay, packet loss rate, and bandwidth utilization rate; the real-time data information is input into a pre-trained reinforcement learning model, wherein the state space of the reinforcement learning model includes bandwidth resource allocation state, network topology state, and user experience quality state;

[0150] The second unit is used to process the real-time data information based on the reinforcement learning model, extract features through a deep neural network, and use a policy gradient algorithm to generate a bandwidth allocation strategy and a routing strategy in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution; sampling based on the bandwidth allocation probability distribution and the routing decision probability distribution to obtain a decision action, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality indicator and a system resource utilization rate; calculating a policy gradient according to the reward value, and updating the parameters of the policy network through back propagation;

[0151] The third unit is used to adjust the bandwidth allocation parameters and routing table configuration in the network in real time through the software-defined network controller according to the bandwidth allocation strategy and routing strategy, perform differentiated bandwidth allocation for each live broadcast room and optimize the data transmission path. The software-defined network controller establishes edge nodes with nearby access based on the geographical distribution of users, and dynamically adjusts the data transmission path when the network is congested.

[0152] According to a third aspect of the embodiments of the present invention,

[0153] An electronic device is provided, comprising:

[0154] processor;

[0155] a memory for storing processor-executable instructions;

[0156] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0157] A fourth aspect of the embodiments of the present invention is:

[0158] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0159] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A private domain live broadcast bandwidth resource allocation and routing method based on reinforcement learning, characterized in that: include: Collecting real-time data information of multiple live broadcast rooms in a private domain live broadcast network, the real-time data information includes the number of live broadcast room users, user geographical distribution, user terminal type, live broadcast content type, network delay, packet loss rate, and bandwidth utilization; inputting the real-time data information into a pre-trained reinforcement learning model, the state space of the reinforcement learning model includes bandwidth resource allocation status, network topology status, and user experience quality status; The real-time data information is processed based on the reinforcement learning model, features are extracted through a deep neural network, and a bandwidth allocation strategy and a routing strategy are generated by a policy gradient algorithm in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution; sampling based on the bandwidth allocation probability distribution and the routing decision probability distribution to obtain a decision action, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality indicator and a system resource utilization rate; calculating a policy gradient according to the reward value, and updating the parameters of the policy network through back propagation; According to the bandwidth allocation strategy and routing strategy, the bandwidth allocation parameters and routing table configuration in the network are adjusted in real time through the software-defined network controller, and differentiated bandwidth allocation is performed for each live broadcast room and the data transmission path is optimized. The software-defined network controller establishes the nearest access edge node based on the geographical distribution of users, and dynamically adjusts the data transmission path when the network is congested; The real-time data information is processed based on the reinforcement learning model, features are extracted through a deep neural network, and a bandwidth allocation strategy and a routing strategy are generated by using a policy gradient algorithm in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution. The steps include: The state vectors of the bandwidth resource allocation state, network topology state and user experience quality state in the live broadcast network are input into the feature importance evaluation layer, and the state vectors are converted into state features through feature engineering. The feature importance evaluation layer constructs a correlation matrix between state features and network performance indicators based on historical decision data, analyzes the influence of each state feature on network performance based on the correlation matrix to obtain feature importance weights, and optimizes the feature importance weights according to real-time decision feedback information. The input state features are adaptively selected based on the optimized feature importance weights, and the selected state features are input into the input layer of the multi-layer perceptron. The hidden layer of the multi-layer perceptron compresses the state features through the ReLU activation function and obtains a low-dimensional compressed feature vector through residual connection; The low-dimensional compressed feature vector is input into the spatiotemporal attention module, the spatial attention submodule of the spatiotemporal attention module obtains the spatial dimension attention weight by calculating the similarity of the state features of different network nodes and links, and the temporal attention submodule of the spatiotemporal attention module obtains the temporal dimension attention weight by calculating the correlation between the feature vectors of different time steps and the current state; the spatial dimension attention weight and the temporal dimension attention weight are respectively weightedly fused with the corresponding features to obtain an enhanced feature representation, and the enhanced feature representation is input into the long short-term memory network unit, the long short-term memory network unit comprises a forget gate, an input gate and an output gate, wherein the forget gate is used to control the degree of forgetting of historical state information, the input gate is used to control the degree of input of current state information, and the output gate is used to control the degree of output of the memory unit state; the long short-term memory network unit extracts the temporal features of the low-dimensional compressed feature vector based on the gate structure and outputs the hidden state; The hidden state is input into a two-layer strategy network, which includes a bandwidth allocation layer and a routing decision layer, wherein the bandwidth allocation layer generates a bandwidth allocation probability distribution through a fully connected structure and a Softmax function, and the routing decision layer generates a routing decision probability distribution through a fully connected structure and a Softmax function.

2. The method according to claim 1, characterized in that The steps of training the reinforcement learning model include: Construct an experience replay pool to store training samples. The training samples are triples including a state vector, an action vector, and a reward value. The state vector includes the bandwidth resource allocation state, the network topology state, and the user experience quality state. The action vector includes the bandwidth allocation decision and the routing decision. The reward value is calculated by the user experience quality index and the system resource utilization index. Collect historical decision data to pre-process the state vector to obtain state features, construct a correlation matrix between the state features and network performance indicators, and obtain feature importance weights by performing singular value decomposition on the correlation matrix; construct a multi-layer perceptron autoencoder with residual connections according to the feature importance weights, compress the state vector into a low-dimensional compressed feature vector and reconstruct it, and train the multi-layer perceptron autoencoder by minimizing the weighted sum of the reconstruction error and the sparse regularization term; Randomly sampling a batch of training data from the experience replay pool, calculating the log-likelihood probability of the current strategy based on the low-dimensional compressed feature vector and the action vector, multiplying the log-likelihood probability by the reward value to obtain the strategy objective function, and updating the strategy network parameters by maximizing the expected value of the strategy objective function; setting the target network, calculating the parameter update amount of the target network by using momentum accumulation, and adaptively adjusting the parameter update step size based on the current training loss; The strategy entropy value is calculated according to the strategy distribution, the exploration probability is adaptively adjusted based on the strategy entropy value, and the reward value is smoothed by short-term and long-term double-scale exponential moving average; the state importance score is calculated based on the feature importance weight, the linear fusion weight of the user experience quality index and the system resource utilization index is dynamically adjusted according to the state importance score, the linear fusion weight is projected into the preset weight range for normalization, and the smoothed reward value is stored again in the experience replay pool.

3. The method according to claim 1, characterized in that The steps of obtaining a decision action based on the bandwidth allocation probability distribution and the routing decision probability distribution sampling, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality index and a system resource utilization rate include: Importance sampling is performed on the bandwidth allocation probability distribution to obtain a candidate bandwidth allocation set, and routing decision sampling is performed on each bandwidth allocation scheme in the candidate bandwidth allocation set in combination with the routing decision probability distribution using the Gumbel-Softmax reparameterization technology to obtain a routing decision matching each bandwidth allocation scheme, thereby forming a decision action pair set; For each group of decision actions in the decision action pair set, an end-to-end delay evaluation model is constructed to calculate the transmission delay, a video quality evaluation function is established to evaluate the impact of bandwidth allocation on video bit rate and clarity, and a packet loss rate detection function is used to evaluate network transmission reliability. The evaluation results of transmission delay, video quality, and network transmission reliability are normalized and weightedly integrated to obtain a user experience quality indicator; Based on each group of decision actions in the decision action pair set, link bandwidth utilization, routing path utilization and load balancing degree are calculated, and the link bandwidth utilization, routing path utilization and load balancing degree are normalized and weightedly integrated to obtain system resource utilization; Normalizing the user experience quality index and system resource utilization, calculating the fluctuation degree of the user experience quality index and the system resource utilization based on the historical decision effect, dynamically adjusting the adaptive fusion weight coefficient according to the fluctuation degree, and weighting the normalized user experience quality index and system resource utilization according to the adaptive fusion weight coefficient to obtain the original reward value; The original reward value is subjected to exponential moving average to obtain a smoothed reward value, a final decision action is selected based on the smoothed reward value using a decreasing exploration probability strategy, and the final decision action and its corresponding smoothed reward value are stored in an experience replay pool for policy network training.

4. The method according to claim 3, characterized in that Importance sampling is performed on the bandwidth allocation probability distribution to obtain a candidate bandwidth allocation set, and routing decision sampling is performed on each bandwidth allocation scheme in the candidate bandwidth allocation set in combination with the routing decision probability distribution using the Gumbel-Softmax reparameterization technology to obtain a routing decision matching each bandwidth allocation scheme. The steps of forming a decision action pair set include: Adaptive importance sampling is performed on the bandwidth allocation probability distribution, wherein the adaptive importance sampling includes calculating the importance weight of the bandwidth allocation scheme, wherein the importance weight is composed of the current state quality score and the historical decision effect score, wherein the historical decision effect score is calculated based on the historical success rate and the volatility of the decision value, and the weight coefficients of the historical success rate and the volatility of the decision value are dynamically adjusted according to the system stability requirements; Adopting a hierarchical attention mechanism to optimize and screen the bandwidth allocation schemes of adaptive importance sampling, calculating the attention weight of the bandwidth resource dimension, reordering the bandwidth allocation schemes based on the attention weight of the bandwidth resource dimension, and selecting multiple bandwidth allocation schemes with the highest scores to form a candidate bandwidth allocation set; Based on each bandwidth allocation scheme in the candidate bandwidth allocation set, the routing decision space is divided into multiple subspaces, the Gumbel-Softmax reparameterization technology is used in each subspace to sample the routing decision, the temperature parameter of each subspace is updated through adaptive learning, and the sampling results of each subspace are merged to obtain the routing decision corresponding to each bandwidth allocation scheme; Perform dynamic quality constraint checks on decision pairs consisting of bandwidth allocation plans and routing decisions, calculate the matching degree of bandwidth and routing and the degree to which the decision pairs satisfy network constraints, dynamically adjust the quality constraint threshold based on the historical decision quality change trend, and retain the decision pairs that meet the quality constraint threshold requirements to form the final decision action pair set.

5. The method according to claim 1, characterized in that According to the bandwidth allocation strategy and routing strategy, the bandwidth allocation parameters and routing table configuration in the network are adjusted in real time through the software-defined network controller, and differentiated bandwidth allocation is performed on each live broadcast room and the data transmission path is optimized. The software-defined network controller establishes the nearest access edge node based on the user's geographical distribution, and dynamically adjusts the data transmission path when the network is congested. The steps include: Deploy a hierarchical software-defined network controller architecture in the network, wherein the hierarchical software-defined network controller architecture includes a central controller and multiple regional controllers, wherein the central controller and the regional controllers perform state synchronization via an east-west interface, and a consistent hashing algorithm is used between the regional controllers for load balancing; Based on the user's geographical distribution information, a clustering algorithm is used to determine the deployment location of the edge node, and the computing resource configuration of the edge node is dynamically adjusted according to the user density; according to the bandwidth allocation strategy output by the reinforcement learning model, the bandwidth allocation quota of each live broadcast room is calculated, and the bandwidth limit field of the flow table is configured based on the bandwidth allocation strategy, which is sent to the network device through the controller, and the bandwidth allocation parameters are adjusted through the exponential weighted moving average algorithm; a weighted directed graph is constructed based on the routing strategy output by the reinforcement learning model, and the node weight of the weighted directed graph is calculated according to the processor load, memory usage and queue length, and the link weight of the weighted directed graph is calculated according to the bandwidth utilization, delay and packet loss rate; Based on the weighted directed graph, the shortest path algorithm considering node weights and link weights is used to calculate the optimal routing path; based on the weighted directed graph, the software-defined network controller monitors the network link status in real time to obtain a congestion index, which is calculated based on queue length, delay and packet loss rate. When the congestion index exceeds a preset threshold, a set of alternative paths is calculated, and the optimal alternative path is selected after evaluating the performance indicators of each alternative path; The path migration is performed in the way of establishing first and then switching. After the new path is established and the corresponding flow table is issued, the traffic is gradually migrated. After the traffic is completely migrated, the original path flow table is deleted. The survival status of the regional controller is detected through heartbeat messages. When the failure of the regional controller is detected, the election algorithm is triggered to select a new main controller, restore the control plane status and rebuild the network view, and recalculate and issue the flow table configuration corresponding to the bandwidth allocation and routing strategy.

6. The method according to claim 5, characterized in that The steps of calculating the optimal routing path using the shortest path algorithm considering node weights and link weights include: The node weight and the link weight are respectively normalized and the comprehensive weight is calculated to obtain the node cost and the link cost; a path cost calculation model is constructed based on the node cost and the link cost, and the path cost is the sum of all node costs and link costs on the path; The improved shortest path algorithm is used to calculate the routing path, including: when performing the path relaxation operation, the cumulative cost from the target node to the source node, the current node cost and the current link cost are weighted and combined based on the dynamic programming principle; a relaxation counter is set to record the number of relaxation times of each node, and when the number of relaxation times of a node exceeds the preset number, the node is temporarily skipped by using the backoff mechanism; a suboptimal path set is maintained during the path search process, and when the optimal path does not meet the constraint conditions, an alternative path is selected from the suboptimal path set; Calculate the relative cost difference between the new and old paths, and trigger a path update operation when the relative cost difference exceeds a preset threshold; establish a new routing path based on the path update operation and issue a corresponding flow table configuration, and evaluate the end-to-end delay, load balancing and path stability of the routing path; dynamically update the comprehensive weight according to the evaluation result, and use the updated comprehensive weight for subsequent path cost calculations.

7. A private domain live broadcast bandwidth resource allocation and routing system based on reinforcement learning, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to collect real-time data information of multiple live broadcast rooms in the private domain live broadcast network, wherein the real-time data information includes the number of users in the live broadcast room, the geographical distribution of users, the type of user terminals, the type of live broadcast content, network delay, packet loss rate, and bandwidth utilization rate; the real-time data information is input into a pre-trained reinforcement learning model, wherein the state space of the reinforcement learning model includes bandwidth resource allocation state, network topology state, and user experience quality state; The second unit is used to process the real-time data information based on the reinforcement learning model, extract features through a deep neural network, and use a policy gradient algorithm to generate a bandwidth allocation strategy and a routing strategy in combination with bandwidth resource constraints, wherein the policy gradient algorithm includes: constructing a two-layer neural network as a policy network, the input layer of the policy network receives the state value in the state space, the hidden layer uses LSTM units to extract time series features, and the output layer generates a bandwidth allocation probability distribution and a routing decision probability distribution; sampling based on the bandwidth allocation probability distribution and the routing decision probability distribution to obtain a decision action, and calculating a reward value after the decision action is executed, wherein the reward value is composed of a weighted value of a user experience quality indicator and a system resource utilization rate; calculating a policy gradient according to the reward value, and updating the parameters of the policy network through back propagation; The third unit is used to adjust the bandwidth allocation parameters and routing table configuration in the network in real time through the software-defined network controller according to the bandwidth allocation strategy and routing strategy, perform differentiated bandwidth allocation for each live broadcast room and optimize the data transmission path. The software-defined network controller establishes edge nodes with nearby access based on the geographical distribution of users, and dynamically adjusts the data transmission path when the network is congested.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images