Multi-objective reinforcement learning method and system for adaptive network flow optimization
Through the multi-objective reinforcement learning method of adaptive network traffic optimization, using meta-network and context-aware strategy generators, combined with federated learning and knowledge distillation, the problem that network traffic optimization methods in the existing technology are difficult to adapt to complex dynamic environments, and realizes the problem of network traffic optimization methods being difficult to adapt to complex dynamic environments. Quick adaptation and efficient optimization improve network performance and resource utilization efficiency.
Patent Information
- Application Number
- CN202411411500.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-12-03
AI Technical Summary
Existing network traffic optimization methods are difficult to adapt to complex dynamic network environments, cannot quickly adapt to burst traffic and emerging applications, and are difficult to achieve global optimization in large-scale distributed networks, and lack generalization capabilities and data privacy protection.
Adopting multi-objective reinforcement learning method with adaptive network traffic optimization, capture multi-time-scale traffic dynamics through meta-network, introduce context-aware meta-strategy generator and federated autonomous learning architecture, combining a priority experience replay algorithm with replay cache and multiple Layer hierarchical timing abstract mechanism, design a distributed collaborative learning framework and knowledge distillation method to achieve rapid adaptation and collaborative optimization.
It significantly improves network performance and resource utilization efficiency, shortens the convergence time of traffic optimization and the adaptation time of abnormal traffic, improves energy efficiency, and realizes distributed collaborative optimization under the premise of protecting privacy.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of network traffic optimization, and in particular to a multi-objective reinforcement learning method and a system thereof for adaptive network traffic optimization. Background Art
[0002] With the rapid development of Internet technology and the increasing complexity of network applications, network traffic optimization has become a key challenge to ensure the quality of network services. Traditional network traffic optimization methods mainly rely on static rules and fixed thresholds. Although they can play a certain role in a stable network environment, their limitations are increasingly prominent in the face of today's complex and changing network traffic patterns. These methods are often difficult to adapt to burst traffic, emerging applications, and dynamically changing network topologies, resulting in low utilization of network resources and reduced user experience quality.
[0003] In recent years, with the advancement of artificial intelligence technology, some researchers have begun to try to apply machine learning, especially reinforcement learning, to network traffic optimization. These methods can improve the optimization effect to a certain extent by learning network behavior patterns. However, existing solutions based on a single reinforcement learning algorithm still have many shortcomings. First, they usually require a large amount of training data and time to achieve ideal performance, which is unacceptable in a rapidly changing network environment. Second, these methods can often only optimize for specific network scenarios, lack generalization capabilities, and are difficult to cope with diverse network environments and service requirements. In addition, in large-scale distributed networks, how to achieve global optimization while protecting data privacy is also an urgent problem to be solved.
[0004] These limitations of existing technologies have severely restricted the further improvement of network performance. With the rapid development of emerging technologies such as cloud computing, 5G, and the Internet of Things, network traffic has become more complex and dynamic than ever before. Traditional methods and simple machine learning algorithms can no longer meet the optimization needs in such a complex environment. Therefore, there is an urgent need for new network traffic optimization methods that can quickly adapt, comprehensively optimize, and collaboratively learn. Summary of the invention
[0005] The purpose of the present invention is to provide a multi-objective reinforcement learning method and system for adaptive network traffic optimization, so as to solve the problem that the network traffic optimization method in the prior art is difficult to adapt to the complex dynamic network environment.
[0006] The present invention provides a multi-objective reinforcement learning method for adaptive network traffic optimization, comprising the following steps: step 1. using a meta-network to capture the laws of adaptive network traffic optimization tasks under different network environments and resource configurations; in order to capture the dynamics of network traffic at multiple time scales, a hierarchical temporal abstraction is established, and a meta-network is constructed based on an autoencoder to learn the laws of traffic at different scales, thereby enhancing the ability of the method in long-term planning; step 2. introducing a context-aware meta-strategy generator to learn the mapping relationship between different scenario requirements and the trained meta-network, thereby achieving rapid adaptation and adjustment to unknown scenarios to improve network performance; establishing an adaptive meta-learning framework to update the strategy; step 3. constructing collaborative optimization between multiple autonomous systems or networks in a federated autonomous learning architecture.
[0007] Specifically, the meta-network and policy table are implemented in a reinforcement learning framework using a prioritized experience replay algorithm with a replay cache to learn step 1 and the learning strategy. The meta-network is used to capture traffic patterns in different dynamic traffic scenarios to characterize the learned and trained network traffic QoS parameters. A multi-layer hierarchical timing abstraction mechanism is designed to calculate traffic indicators in a low aggregation time, thereby improving the meta-network encoder and decoder's ability to extract feature information representation of traffic.
[0008] Specifically, the QoS configuration is obtained by weighted summation.
[0009]
[0010] Among them, QoS overall service quality configuration, For the The weight of each QoS parameter, For the The value of QoS parameters; a single meta-network and a policy table are used to jointly model QoS parameters. The reinforcement learning environment first provides traffic data patterns or resource configurations as inputs for training the meta-network; in order to capture multi-scale dynamic traffic patterns, fine-grained and medium-scale QoS objectives are adopted; in step 2, the context-aware policy generator uses meta-policies to optimize the weight allocation of target QoS parameters; the context-aware policy network first collects QoS parameter data of traffic to represent the current state, then it generates a set of policy parameter-based weights in the best meta-network by querying the meta-network, and finally generates a set of optimized QoS weight parameters according to the given QoS parameters at multiple scales; the policy table is represented by a set of QoS hyperparameters, which assign different priorities to QoS parameters and learn QoS configuration schemes;
[0011]
[0012] in, For the The value of a QoS parameter, Strategy generation algorithm, For the indivual The hyperparameters of the parameters, The current network context; based on the policy generation algorithm, the QoS hyperparameters are adjusted to obtain the QoS parameter values; based on the QoS parameter values, the model generates traffic configuration according to actual needs; and the traffic configured in the policy table is implemented.
[0013] Specifically, the meta-algorithm and autoencoder are combined for meta-network training based on the policy gradient optimization method; in step 2, the meta-learning agent updates the policy table according to the environment interaction, and designs an accumulated experience replay storage buffer fixed to the scene to collect experience data for training the meta-network. The meta-experience composed of a series of states, actions and rewards of old scenes continuously reduces the difference between the sampled data distribution and the target distribution by caching and replaying the data from the trained meta-network; the old environment is adapted through the context-aware policy network, and a meta-experience replay storage buffer fixed to the scene accumulation is designed to interact with the meta-agent environment with different strategies; the transfer meta and policy meta tables are used to interact in the old environment.
[0014] Specifically, a distributed collaborative learning framework is designed to encourage cooperation between multiple autonomous systems or network domains. A unified supervisory signal is generated based on the global QoS goal to guide the policy adjustment. In order to avoid the additional overhead of frequent communication between networks, a federated learning-based method is adopted to mine the common information between multiple autonomous systems or multiple network domains. Each domain updates the policy locally according to its global weight.
[0015] Specifically, each meta-network is updated and trained through the contextual attention mechanism. The meta-learning agent uses a shared meta-strategy to generate a combination of randomly sampled strategies when generating the state of the current scene to improve the generalization ability of the strategy. The contextual attention mechanism uses the following algorithm:
[0016]
[0017] Among them, a i For the The attention weight of each element, is the learnable parameter matrix, For the The hidden state of an element, is the query vector;
[0018] A set of different strategies are sampled from all strategy vectors, and an independent attention mechanism weight is assigned to the additional space vector in the state of the generated meta-learning agent to obtain the strategy corresponding to the current environment; the environment state is generated by the network operation indicators composed of the bottleneck resources, network node bandwidth, latency, throughput and traffic ratio of the current network flow; when a multi-objective optimization method is adopted, the meta-learning agent dynamically adjusts the learning rate of each meta-network based on the difference in the objective function value using game theory. The dynamic learning rate adjustment formula is as follows:
[0019]
[0020] in, For the The learning rate of the meta-network, is the basic learning rate, For the The value of the objective function, To adjust the coefficient, control the sensitivity of changes in the learning rate.
[0021] Specifically, the federated learning weight that should be assigned to each autonomous system or network domain is estimated through the contribution evaluation algorithm based on the Shapley value, and common supervision signals are dynamically allocated to different autonomous systems in the federated learning framework to ensure that the objective function can converge; according to the Shapley value formula, each policy table will obtain a federal weight to ensure that the information allocation to all policy tables is correctly adjusted; the Shapley value evaluation determines the attention distribution of each network or autonomous system, which is used to share policy information to ensure that the policy training obtains good performance. The Shapley value calculation formula is:
[0022]
[0023] in, Player The Shapley value of is the set of all players, Does not include players A subset of Characteristic function, which represents the value of the alliance.
[0024] Specifically, the policy table uses a logarithmic function to obtain the distribution; based on the Shapley value, the sharing ratio of the global supervision signal is: For each strategy, federated learning training round, network domain, and time period, Representative Moment Network domain in The Shapley value of federated learning training rounds; is the number of learning rounds; all policy tables receive the global supervision signal shared by the federation; then the policy parameters are regenerated, the target is updated, and the policy parameters are optimized. In the federated learning training mode, each domain updates the local policy according to its global weight. The policy update formula is: ,in, is the policy function, For the current strategy, the method is adjusted continuously over several cycles to obtain the global goal. After the formula is converted, it is obtained: , For the moment The strategy parameters, Learning Rate is the policy gradient, For the status Next select action The probability of For the state-action value function, gradient approximation is introduced to calculate it, and the gradient is obtained through the small batch strategy table; the training rounds of the network environment are traversed, and data is collected iteratively. When m is large enough, the gradient approximation calculation when each small batch is a single strategy table replaces the global gradient calculation.
[0025] Specifically, a knowledge distillation method is used to provide sharing of experience information between network regions or autonomous systems; each policy table is assigned a weight, and some policy tables are randomly selected from them to generate similar experience data in a given replay memory.
[0026] The system for executing the multi-objective reinforcement learning method for adaptive network traffic optimization includes: a meta-network training module, which is used to use the meta-network to capture the laws of adaptive network traffic optimization tasks under different network environments and resource configurations; in order to capture the dynamics of network traffic at multiple time scales, a meta-network is constructed to learn the laws of traffic at different scales, and enhance the ability of the method in long-term planning; a context-aware meta-strategy generator module, including a meta-strategy network training module and a strategy network update module; the meta-strategy network training module includes: a context-aware strategy generator to enhance the strategy table for quickly adapting to different network environments and user demand scenarios; the adaptive meta-learning module includes the above The following perception strategy generator and strategy table learning module are used to update the strategy table. Different learning rates are designed for different meta-tables during strategy table training. Each strategy table has the same impact on the overall objective function. Based on the objective function value of the strategy, the learning rate of each meta-table is calculated using game theory formulas. The federated learning module is used to build collaborative optimization between multiple autonomous systems or networks in a federated autonomous learning architecture, build a distributed collaborative learning framework, and realize collaborative learning of multi-characteristic network regions. The contribution algorithm based on Shapley value is introduced to dynamically adjust the weights of different network regions or autonomous systems in the learning process, and introduce knowledge distillation methods to improve the mobility and adaptability of network models. Advantages and beneficial effects of the present invention:
[0027] Compared with the existing technology, the present invention provides a new multi-objective reinforcement learning method and system for adaptive network traffic optimization by introducing advanced technologies such as meta-learning, context awareness, and federated learning. The method can quickly adapt to different network environments, achieve dynamic balance of multiple objectives, and realize distributed collaborative optimization under the premise of protecting privacy, thereby significantly improving network performance and resource utilization efficiency.
[0028] The implementation of the present invention can not only significantly improve the key performance indicators of the network such as end-to-end delay, bandwidth utilization and throughput, but also significantly shorten the convergence time of traffic optimization and the adaptation time of abnormal traffic. At the same time, through more efficient resource scheduling and energy management, the present invention can also achieve better energy efficiency. These advantages make the present invention have broad application prospects in complex scenarios such as modern enterprise networks, cloud computing data centers, 5G mobile networks, etc., and provide an innovative and efficient solution to the challenges faced by current network traffic optimization. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a logic block diagram of the system of the present invention.
[0030] Figure 2 It is a logic block diagram of the meta-network training module of the present invention.
[0031] Figure 3It is a logical block diagram of the context-aware meta-strategy generator module of the present invention.
[0032] Figure 4 It is a logical block diagram of the adaptive meta-learning module of the present invention.
[0033] Figure 5 This is a logical block diagram of the federated learning module of the present invention.
[0034] Figure 6 It is a logic block diagram of the contribution algorithm method based on Shapley value of the present invention.
[0035] Figure 7 It is a logical block diagram of the knowledge distillation method of the present invention. DETAILED DESCRIPTION
[0036] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0037] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0038] Example 1
[0039] See also Figure 1-7 The multi-objective reinforcement learning method for adaptive network traffic optimization provided in this embodiment includes the following steps:
[0040] Step 1. Use a meta-network to capture the rules of adaptive network traffic optimization tasks under different network environments and resource configurations; in order to capture the dynamics of network traffic at multiple time scales, a hierarchical temporal abstraction is established, and a meta-network is constructed based on an autoencoder to learn the rules of traffic at different scales, enhancing the ability of the method in long-term planning;
[0041] Step 2. Introduce a context-aware meta-strategy generator to learn the mapping relationship between different scenario requirements and the trained meta-network, and achieve rapid adaptation to unknown scenarios to improve network performance; establish an adaptive meta-learning framework to update the strategy;
[0042] Step 3. Build collaborative optimization among multiple autonomous systems or networks of a federated autonomous learning architecture.
[0043] The present invention proposes an innovative adaptive network traffic optimization method. First, a meta-network is used to capture the laws of network traffic optimization tasks in different environments. This method can adapt to various network conditions, thereby improving the universality of the optimization effect. For example, in a large enterprise network, different departments may have different network requirements and characteristics. By using a meta-network, the optimization strategies for high-frequency small data packet transmission in the sales department and low-frequency large data packet transmission in the R&D department can be learned at the same time.
[0044] Secondly, in order to capture the dynamics of network traffic at multiple time scales, a hierarchical temporal abstraction is established. This approach can simultaneously consider short-term, medium-term, and long-term changes in network traffic. For example, in the network of an e-commerce platform, the impact of user click behavior at the second level, shopping peak hours at the hour level, and seasonal sales activities on network traffic can be considered simultaneously. Through this multi-scale consideration, a more comprehensive and forward-looking optimization strategy can be formulated.
[0045] Then, a context-aware meta-policy generator is introduced. The advantage of this innovation is that it can quickly adapt to unknown network scenarios. For example, when a new application suddenly becomes popular and generates a lot of traffic, the system can quickly adjust the policy to adapt to this change without retraining the entire model.
[0046] Finally, a federated autonomous learning architecture is constructed to achieve collaborative optimization among multiple autonomous systems or networks. The beneficial effect of this approach is that it allows different network systems to learn collaboratively while protecting privacy. For example, multiple enterprises can jointly optimize their network performance without directly sharing sensitive network data.
[0047] Using the prioritized experience replay algorithm with replay cache, the meta-network and policy table are implemented in the reinforcement learning framework to learn step 1 and the learning strategy. The meta-network is used to capture the traffic patterns in different dynamic traffic scenarios to characterize the learned and trained network traffic QoS parameters. A multi-layer hierarchical timing abstraction mechanism is designed to calculate traffic indicators in a low aggregation time, thereby improving the meta-network encoder and decoder's ability to extract feature information representation of traffic.
[0048] The present invention adopts a priority experience replay algorithm with replay buffer. The advantage of this method is that it can more effectively utilize historical experience and accelerate the learning process. Specifically, those experiences with high TD errors can be replayed preferentially, which usually contain more valuable information.
[0049] In practical applications, you can set an experience pool of size 100,000 and use the following formula to calculate the probability of each experience being sampled:
[0050]
[0051] in, is the probability that the ith experience is sampled, is the TD error, is a small positive number (e.g. 0.01), is the priority index (usually set to 0.6). This method can significantly improve learning efficiency, especially when processing large-scale network traffic data.
[0052] In addition, a multi-layer hierarchical temporal abstraction mechanism is designed, which can calculate traffic indicators in low-aggregation time, thereby improving the ability of the meta-network to extract traffic features. For example, three time scales can be set: 5 seconds, 1 minute, and 10 minutes, respectively capturing short-term, medium-term, and long-term traffic features. This multi-scale feature extraction can help to more comprehensively understand the dynamic characteristics of network traffic.
[0053] The QoS configuration is obtained by weighted summation. A single meta-network and a policy table are used to jointly model the QoS parameters. The reinforcement learning environment first provides traffic data patterns or resource configurations as inputs for training the meta-network. In order to capture multi-scale dynamic traffic patterns, fine-grained and medium-scale QoS objectives are adopted. In step 2, the context-aware policy generator uses meta-policies to optimize the weight distribution of target QoS parameters. The context-aware policy network first collects QoS parameter data of the traffic to represent the current state. Then, it generates a set of policy parameter-based weights in the best meta-network by querying the meta-network. Finally, a set of optimized QoS weight parameters are generated according to the given QoS parameters at multiple scales.
[0054] The present invention adopts a weighted summation method to obtain QoS configuration. The advantage of this method is that it can flexibly adjust the importance of different QoS parameters. For example, in a video streaming application, bandwidth and delay may be given higher weights, while in a file transfer application, throughput may be more emphasized.
[0055] Specifically, the overall QoS can be calculated using the following formula:
[0056]
[0057] in, is the weight of the i-th QoS parameter, is the value of the i-th QoS parameter. In practical applications, these weights can be dynamically adjusted according to network conditions and application requirements.
[0058] In addition, the present invention adopts fine-grained and mesoscale QoS objectives to capture multi-scale dynamic traffic patterns. The benefit of this approach is that it can consider both short-term fluctuations and medium-term trends, thereby formulating a more robust optimization strategy. For example, the instantaneous delay can be monitored at the 1-second level (fine-grained), while the average throughput can be monitored at the 1-minute level (mesoscale).
[0059] The policy table is represented by a set of QoS hyperparameters. The hyperparameters assign different priorities to QoS parameters and learn QoS configuration schemes. The QoS hyperparameters are adjusted based on the policy generation algorithm to obtain QoS parameter values. Based on the QoS parameter values, the model generates traffic configuration according to actual needs. The traffic configured in the policy table is implemented.
[0060] The present invention uses QoS hyperparameters to represent the policy table. The advantage of this method is that it can adjust the priority of QoS parameters more flexibly. For example, in a real-time communication application, the delay parameter may be given a higher priority, while in a large file transfer application, the throughput may be given priority.
[0061] Specifically, the QoS parameter value can be calculated using the following formula:
[0062]
[0063] in, is the value of the i-th QoS parameter, f is the policy generation algorithm, is the hyperparameter of the i-th QoS parameter, and c is the current network context. In practical applications, a neural network can be used to implement function f, whose input is the hyperparameter and network context, and the output is the specific QoS parameter value.
[0064] The beneficial effect of this approach is that it can dynamically adjust QoS parameters according to actual network conditions and application requirements, thereby achieving more refined network traffic optimization.
[0065] Based on the policy gradient optimization method, the meta-algorithm and autoencoder are combined for meta-network training; in step 2, the meta-learning agent updates the policy table according to the environment interaction, and designs an accumulated experience replay storage buffer fixed to the scene to collect experience data for training the meta-network. The meta-experience consists of a series of states, actions and rewards of old scenes. By caching and replaying the data from the training meta-network, the difference between the sampled data distribution and the target distribution is continuously reduced.
[0066] The present invention combines the policy gradient optimization method with the autoencoder for meta-network training. The advantage of this method is that it can simultaneously learn an effective representation of the network state and an optimization strategy. For example, in a complex enterprise network, the autoencoder can learn a low-dimensional representation of key features such as network topology and traffic patterns, while the policy gradient method optimizes the decision-making strategy based on these representations.
[0067] Specifically, the following formula can be used to update the policy parameters:
[0068]
[0069] Among them, θ is the policy parameter, α is the learning rate, J(θ) is the policy objective function, is the policy gradient. In practical applications, the Actor-Critic architecture can be used to estimate the policy gradient, which can reduce variance and improve learning stability.
[0070] In addition, the present invention designs an accumulated experience playback storage buffer fixed to the scene. The beneficial effect of this method is that it can effectively utilize historical experience and accelerate the learning process. For example, an experience pool of 500,000 can be set, and batch data can be selected from it for training using random sampling. This method can break the time correlation between data, thereby improving the efficiency and stability of learning.
[0071] Adapt to the old environment through the context-aware policy network, design a meta-experience replay storage buffer fixed to the scene accumulation, interact with the environment with different policy meta-agents; use transfer meta and policy meta-table to interact in the old environment.
[0072] The present invention adapts the old environment through a context-aware policy network. The advantage of this method is that it can quickly adapt to the new network environment without having to learn from scratch. For example, when a company acquires another company and needs to integrate the network, the system can quickly adapt to the new network structure based on the understanding of the old environment.
[0073] Specifically, the metaexperience replay buffer can be updated using the following formula:
[0074]
[0075] in, is the meta-experience replay buffer at time t, It's the state. It's action. It's a reward. is the next state. In practical applications, a fixed-size buffer (e.g., 1,000,000 experiences) can be set, and when the buffer is full, a first-in, first-out (FIFO) strategy is used to replace the old experiences.
[0076] The beneficial effect of this approach is that it maintains the system's memory of historical experience while constantly absorbing new experience, thereby achieving continuous learning and adaptation.
[0077] A distributed collaborative learning framework is designed to encourage cooperation between multiple autonomous systems or network domains. A unified supervisory signal is generated based on the global QoS goal to guide the policy adjustment. In order to avoid the additional overhead of frequent communication between networks, a federated learning-based method is adopted to mine the common information between multiple autonomous systems or multiple network domains. Each domain updates the policy locally according to its global weight.
[0078] The present invention designs a distributed collaborative learning framework. The advantage of this method is that it can achieve effective cooperation between multiple autonomous systems or network domains while protecting the data privacy of each system. For example, in a multinational enterprise network, branches in various countries can jointly optimize the overall network performance without directly sharing the original network data that may involve privacy.
[0079] Specifically, a federated learning-based approach is used to achieve this collaboration. In each round of learning, each participant (such as a branch in each country) first updates the model locally with its own data, and then sends only the model update (not the original data) to the central server. The central server aggregates these updates and then distributes the aggregated model to all participants. This process can be expressed by the following formula:
[0080]
[0081] in, is a global model parameter, is the data volume of the ith participant, n is the total data volume, are the local model parameters of the ith participant.
[0082] This method can improve the overall model performance by utilizing the data of all participants while protecting privacy. In practical applications, global aggregation can be performed every 1,000 local training steps, which can achieve a good balance between communication overhead and model performance.
[0083] Each meta-network is updated and trained through the contextual attention mechanism. The meta-learning agent uses a shared meta-strategy to generate a combination of randomly sampled strategies when generating the state of the current scene to improve the generalization ability of the strategy; a set of different strategies are sampled from all strategy vectors, and an independent attention mechanism weight is assigned to the additional spatial vector in the state of the generated meta-learning agent to obtain the strategy corresponding to the current environment.
[0084] The present invention introduces a contextual attention mechanism to update and train the meta-network. The advantage of this method is that it can dynamically focus on the most relevant information based on the characteristics of the current network environment. For example, in a complex data center network, the system may pay more attention to load balancing-related features during high-load periods and more attention to energy efficiency-related features during low-load periods.
[0085] Specifically, the attention weight can be calculated using the following formula:
[0086] a i
[0087] Among them, a i is the attention weight of the ith element, v, W, U are learnable parameter matrices, is the hidden state of the ith element, and q is the query vector (representation of the current environment).
[0088] The benefit of this method is that it can adaptively adjust the attention paid to different features, thereby making more accurate and effective decisions. In practical applications, a multi-head attention mechanism can be used, for example, 8 attention heads are set, each with a dimension of 64, so that multiple different feature combinations can be paid attention to at the same time.
[0089] The environmental state is generated by network operation indicators consisting of the bottleneck resources, network node bandwidth, latency, throughput and flow ratio of the current network flow; when a multi-objective optimization method is adopted, the meta-learning agent dynamically adjusts the learning rate of each meta-network using game theory according to the difference in objective function values.
[0090] The present invention considers multiple key network operation indicators to generate the environment status. This comprehensive status representation method can provide a comprehensive view of network performance and help make more accurate optimization decisions. For example, in a large enterprise network, the bandwidth utilization of the core switch, the end-to-end delay of key applications, the overall network throughput, and the proportion of different types of traffic can be monitored simultaneously.
[0091] In addition, the present invention introduces a dynamic learning rate adjustment mechanism based on game theory in multi-objective optimization. The advantage of this method is that it can dynamically allocate computing resources according to the degree of optimization of different objectives. Specifically, the following formula can be used to adjust the learning rate of each meta-network:
[0092]
[0093] in, For the The learning rate of the meta-network, is the base learning rate (e.g. 0.001), For the The value of the objective function, A scaling factor (usually set between 0.5 and 2) that controls the sensitivity of changes in the learning rate.
[0094] The benefit of this approach is that it can strike a good balance between multiple optimization objectives that may conflict with each other. For example, when optimizing network performance, you may need to balance between minimizing latency and maximizing throughput, and this mechanism can dynamically adjust the focus on these two objectives based on the current optimization status.
[0095] The federated learning weight that should be assigned to each autonomous system or network domain is estimated through a contribution evaluation algorithm based on the Shapley value, and a common supervision signal is dynamically allocated to different autonomous systems in the federated learning framework to ensure that the objective function can converge; according to the Shapley value formula, each policy table will obtain a federal weight to ensure that the information distribution of all policy tables is correctly adjusted; the Shapley value evaluation determines the attention distribution of each network or autonomous system, which is used to share policy information to ensure that the policy training achieves good performance.
[0096] The present invention introduces a contribution evaluation algorithm based on Shapley value. The advantage of this method is that it can fairly evaluate the contribution of each participant to the overall performance, thereby more reasonably allocating federated learning weights. For example, in an alliance network composed of multiple enterprises, some enterprises may contribute more to the overall optimization due to their unique network structure or traffic pattern, and therefore should receive a higher weight. The calculation formula of Shapley value is as follows:
[0097]
[0098] in, Player The Shapley value of is the set of all players, Does not include players A subset of Characteristic function, which represents the value of the alliance.
[0099] In practical applications, since the complexity of calculating all possible subsets is too high, the Monte Carlo method can be used for approximate calculation. Specifically, 1,000 permutations can be randomly generated, and then the marginal contribution of each participant can be calculated, and finally the average value is taken as the estimate of the Shapley value.
[0100] This approach can more fairly and effectively utilize the contributions of all participants, thereby improving the overall optimization effect. At the same time, it also provides a reasonable incentive mechanism for participants to actively participate in collaborative optimization.
[0101] It can be seen that the present invention provides a comprehensive, efficient and adaptive network traffic optimization method. This method can achieve a dynamic balance of multiple objectives in a complex and changeable network environment, and improve the overall performance through distributed collaborative learning. Next, other innovative points and implementation details of the present invention will be discussed.
[0102] The policy table uses a logarithmic function to obtain the distribution; based on the Shapley value, the sharing ratio of the global supervision signal is: , s, m, j are each strategy, federated learning training round, network domain, and time period respectively. represents the Shapley value of the mth network domain at time s in the jth federated learning training round; , is the number of learning rounds; all strategy tables receive the global supervision signal shared by the federation; then the strategy parameters are regenerated, the objectives are updated, and the strategy parameters are optimized.
[0103] The present invention adopts a global supervisory signal allocation method based on logarithmic function and Shapley value. The advantage of this method is that it can allocate global supervisory signals more fairly and effectively, thereby promoting the collaborative optimization of the entire system. For example, in a large cloud computing network consisting of multiple data centers, each data center may have different traffic patterns and resource configurations. This method can allocate optimization resources according to the actual contribution of each data center.
[0104] Specifically, the sharing ratio of the global supervision signal The calculation formula uses a logarithmic function. The advantage of this is that it can compress the value range while maintaining the relative order, avoiding excessive influence of extreme values on the results. In practical applications, a logarithmic function with base 2 can be selected, which can better balance the difference between large and small values.
[0105] In addition, the present invention also introduces This formula is used to calculate the number of learning rounds. The benefit of this method is that it can Converted to a probability distribution, which makes it easier to perform subsequent random sampling or weight assignment. In practice, this probability distribution can be used to determine the frequency of participation of each participant in each round of global update.
[0106] In the federated learning training mode, each domain updates its strategy locally according to its global weight. The present invention adopts a local strategy update method based on global weights in the federated learning training mode. This method can fully utilize global information to guide local optimization while protecting data privacy. For example, in a multinational enterprise network, each country branch can adjust the strength of its local optimization strategy according to its importance in the global network (reflected by the global weight).
[0107] Specifically, the local policy can be updated using the following formula:
[0108]
[0109] in, are local model parameters, η is the learning rate, is the global weight, is the gradient of the local loss function.
[0110] This approach enables each participant to adjust the optimization intensity according to its importance in the whole without directly sharing sensitive data. In practical applications, a global weight update can be performed every 100 local training steps, which can achieve a good balance between computational efficiency and optimization effect.
[0111] The strategy update formula is: ,in, is the policy function, For the current strategy, the method is adjusted continuously over several cycles to obtain the global goal. After the formula is converted, it is obtained: , introduce gradient approximation to calculate, and obtain the gradient through the small batch strategy table; traverse the training rounds of the network environment, collect data iteratively, and when m is large enough, the gradient approximation calculation when each small batch is a single strategy table replaces the global gradient calculation.
[0112] The invention uses a policy gradient method to update the policy. The advantage of this method is that it can directly optimize the policy without the need for a value function as an intermediate step, so that it can converge to the optimal policy faster in some cases. For example, in a network traffic scheduling task that requires fine control, the policy gradient method can directly learn the optimal scheduling policy.
[0113] The policy update formula in this invention is the core of the policy gradient algorithm. In practical applications, its conversion form is usually used: Here, α is the learning rate, which is usually set between 0.001 and 0.01. is the probability of choosing action a in state s, is the state-action-value function.
[0114] In order to improve the computational efficiency, the present invention introduces a method of gradient approximation and small batch update. Specifically, a batch size of 32 or 64 can be used to randomly sample a batch of data from the experience replay buffer each time to calculate the gradient. The beneficial effect of this method is that it can significantly improve the computational speed while maintaining the update stability.
[0115] The knowledge distillation method is used to provide sharing of experience information between network areas or autonomous systems. The present invention introduces a knowledge distillation method to achieve sharing of experience information between network areas or autonomous systems. The advantage of this method is that it can achieve effective transfer of knowledge without directly sharing the original data. For example, in a system composed of multiple independent but related networks, the optimization strategy learned in one network can be transferred to other networks through knowledge distillation.
[0116] The core idea of knowledge distillation is to use a teacher model to guide the learning of the student model. In the present invention, the following loss function can be used to implement knowledge distillation:
[0117]
[0118] Among them, H is the cross entropy function, y is the true label, is the output of the student model, The output of the teacher model, T is the temperature parameter (usually set to 2 or 3), and α is the balancing factor (usually set to 0.5).
[0119] This method can achieve knowledge sharing between different network areas or autonomous systems while protecting privacy, thereby improving the performance of the overall system. In practical applications, knowledge distillation can be performed every 1,000 training steps, which can achieve a good balance between computational overhead and knowledge transfer effect.
[0120] Each policy table is assigned a weight, and some policy tables are randomly selected from them that produce similar experience data in a given replay memory.
[0121] The present invention assigns weights to each strategy table and performs random selection based on these weights to generate similar experience data in the playback memory. The advantage of this method is that it can focus more on those strategies that perform better while maintaining diversity. For example, in a complex network traffic management system, there may be multiple strategies optimized for different situations. Through this weighted random selection method, those strategies that perform well in specific situations can be more effectively utilized.
[0122] Specifically, the probability of selecting each strategy table can be calculated using the following formula:
[0123]
[0124] in, is the probability of selecting the i-th strategy table, is the weight of the ith strategy table. The weight can be adjusted dynamically based on the historical performance of the strategy. For example, the average reward of the strategy can be used as the weight.
[0125] The beneficial effect of this method is that it can strike a good balance between exploration and exploitation, thereby accelerating the learning process and improving the final performance. In practical applications, the weights of the strategy table can be updated every 100 training steps, so that the latest performance of each strategy can be reflected in a timely manner.
[0126] Example 2
[0127] Combination Figure 1 - Figure 7 ,An embodiment of the present invention further provides a system for executing the above-mentioned multi-objective reinforcement learning method for adaptive network traffic optimization, including: a meta-network training module 1 for capturing the regularities of adaptive network traffic optimization tasks under different network environments and resource configurations by using a meta-network; in order to capture the dynamics of network traffic at multiple time scales, a meta-network is constructed to learn the regularities of traffic at different scales to enhance the method’s ability in long-term planning;
[0128] The context-aware meta-policy generator module 2 includes a meta-policy network training module and a policy network update module; the meta-policy network training module includes: a context-aware policy generator to enhance the policy table for quickly adapting to different network environments and user demand scenarios;
[0129] The adaptive meta-learning module 3 includes a context-aware policy generator 2 and a policy table learning module 4, which is used to update the policy table and design different learning rates for different meta-tables during policy table training; each policy table has the same impact on the overall objective function; based on the objective function value of the policy, the learning rate of each meta-table is calculated using a game theory formula;
[0130] The federated learning module 5 is used to build collaborative optimization among multiple autonomous systems or networks of a federated autonomous learning architecture, build a distributed collaborative learning framework, and realize collaborative learning in multi-characteristic network regions;
[0131] A contribution algorithm based on Shapley value was introduced6 to dynamically adjust the weights of different network regions or autonomous systems during the learning process, and a knowledge distillation method7 was introduced to improve the mobility and adaptability of the network model.
[0132] The present invention proposes a complete adaptive network traffic optimization system, which includes multiple key modules. The advantage of this system design is that it can comprehensively handle various challenges in network traffic optimization, thereby achieving efficient, flexible and adaptive optimization.
[0133] First, the meta-network training module 1 uses the idea of meta-learning to quickly adapt to different network environments and resource configurations. For example, in a large enterprise network, different departments or branches may have different network characteristics and requirements. The meta-network can quickly adapt to these differences and provide customized optimization strategies. In practical applications, a stacked long short-term memory network (LSTM) can be used to build a meta-network to capture network traffic dynamics at different time scales. For example, a three-layer LSTM can be set up with 128, 64, and 32 neurons in each layer, which can effectively learn short-term, medium-term, and long-term traffic patterns.
[0134] Secondly, the context-aware meta-strategy generator module 2 is able to dynamically generate optimization strategies based on the current network conditions and user needs. This is particularly useful when dealing with burst traffic or new application scenarios. For example, when a new high-bandwidth application suddenly becomes popular, the system can quickly adjust the strategy to adapt to this change. Specifically, an attention mechanism can be used to achieve context awareness. For example, a multi-head attention mechanism can be used, with 8 attention heads and a dimension of 64 for each head, so that multiple different context features can be paid attention to at the same time.
[0135] Thirdly, the adaptive meta-learning module 3 realizes a more flexible and efficient learning process by designing different learning rates for different strategy tables. The beneficial effect of this method is that it can dynamically adjust the learning speed according to the performance of each strategy table, thereby accelerating the overall optimization process.
[0136] The federated learning module 5 enables collaborative optimization between multiple autonomous systems or network regions. This is particularly valuable in large-scale, distributed network environments. For example, in a global network of a multinational enterprise, networks in various regions can collaboratively optimize overall network performance without directly sharing sensitive data. Specifically, a variant of the FedAvg algorithm can be used, and each round of federated learning iteration includes the following steps: the central server selects a portion of participants (for example, the selection rate is 0.3). The selected participants use local data to update the model, and the number of iterations is E (for example, E=5). Participants send model updates to the central server; the central server aggregates the updates; the central server distributes the aggregated model to all participants.
[0137] The contribution algorithm based on Shapley value 6 further improves the fairness and efficiency of the system. By dynamically adjusting the weights of different network areas or autonomous systems during the learning process, the contribution of each participant can be better utilized. The calculation of Shapley value can be approximated using the Monte Carlo method. For example, 1000 permutations can be randomly generated, and then the marginal contribution of each participant is calculated, and finally the average is taken as the estimate of Shapley value.
[0138] Finally, the knowledge distillation method 7 improves the transferability and adaptability of network models. By transferring the knowledge of the best performing model (teacher model) to other models (student models), optimization experience can be efficiently shared between different network regions.
[0139] In general, the adaptive network traffic optimization system provided by the present invention can achieve efficient and flexible optimization in complex and dynamic network environments by combining multiple advanced machine learning technologies, and has broad application prospects. For example, it can be applied to large enterprise networks, cloud computing data centers, 5G mobile networks and other scenarios, significantly improving network performance and user experience.
[0140] Example 3
[0141] In order to verify the effect of the present invention, the adaptive network traffic optimization system of the present invention is applied in the global network of a large multinational enterprise. The network includes multiple data centers and branches distributed in different geographical locations, facing complex and changeable traffic patterns and diverse service quality requirements. A complete system including a meta-network training module 1, a context-aware meta-strategy generator module 2, an adaptive meta-learning module 3, a federated learning module 5, a contribution algorithm method based on Shapley value 6 and a knowledge distillation method 7 is deployed. After the system was run for 30 days, a comprehensive evaluation of the network performance was carried out.
[0142] Comparative Example 1: In the same network environment, a traditional static traffic optimization system is deployed, which uses fixed rules and thresholds to manage network traffic.
[0143] Comparative Example 2: A traffic optimization system based on a single reinforcement learning algorithm is applied in the same network. The system does not have meta-learning and federated learning capabilities.
[0144] In order to comprehensively evaluate the system performance, the following key indicators were selected:
[0145] 1. Average end-to-end latency (milliseconds);
[0146] 2. Bandwidth utilization (%);
[0147] 3. Throughput (Gbps);
[0148] 4. Traffic optimization convergence time (minutes);
[0149] 5. Abnormal traffic adaptation time (seconds);
[0150] 6. Energy efficiency (energy required per bit transmitted, nJ / bit);
[0151] The testing methods for these indicators are as follows:
[0152] 1. Average end-to-end latency: Using distributed network probes, the round-trip time of all critical paths in the network is measured every 5 minutes.
[0153] 2. Bandwidth utilization: Collect traffic data of all core links through the network management system and calculate the ratio of actual traffic to link capacity.
[0154] 3. Throughput: Deploy traffic generators at network edge nodes to simulate actual business loads and measure the data transmission rate of the entire network.
[0155] 4. Traffic optimization convergence time: Introduce a new traffic pattern and record the time required for the system to reach a stable state.
[0156] 5. Abnormal traffic adaptation time: simulate sudden traffic events and measure the time required for the system to adjust its strategy and restore normal performance.
[0157] 6. Energy efficiency: Energy consumption data is collected through the power management module of the network device and calculated based on the overall data transmission volume.
[0158] After 30 days of testing, the following results were obtained:
[0159] From these test results, we can clearly see the outstanding advantages of the present invention:
[0160] First, in terms of average end-to-end latency, the present invention reduces it by 46.7% compared to traditional static systems and 21.9% compared to single reinforcement learning systems. This significant improvement is due to the meta-network training module 1 and the context-aware meta-strategy generator module 2, which can quickly adapt to network conditions and optimize routing strategies.
[0161] Secondly, the present invention performs well in terms of bandwidth utilization and throughput. Compared with comparative example 1, the bandwidth utilization is improved by 26.0% and the throughput is improved by 25.6%. This is mainly due to the adaptive meta-learning module 3, which can continuously optimize the resource allocation strategy.
[0162] It is particularly noteworthy that the present invention performs well in the face of dynamic changes. The flow optimization convergence time is only 8.5 minutes, which is 61.9% faster than the comparative example 2. Even more impressive is that in terms of abnormal flow adaptation time, the present invention can respond in only 3.2 seconds, which is nearly 60 times faster than the traditional static system and nearly 5 times faster than the single reinforcement learning system. This fully demonstrates the powerful performance of the context-aware meta-strategy generator module 2 and the contribution algorithm method 6 based on Shapley value.
[0163] Finally, in terms of energy efficiency, the energy required for each bit of transmission in the present invention is saved by 39.3% compared with comparative example 1 and 24.1% compared with comparative example 2. This is due to the federated learning module 5 and the knowledge distillation method 7, which can share and optimize strategies across the entire network, thereby achieving more efficient resource utilization.
[0164] These test results fully demonstrate the excellent performance of the invention in complex and dynamic network environments. It can not only significantly improve the overall performance of the network, but also quickly adapt to changes and improve energy efficiency while ensuring performance. This has important practical application value for scenarios such as modern enterprise networks, cloud computing data centers and 5G mobile networks.
[0165] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A multi-objective reinforcement learning method for adaptive network traffic optimization, characterized in that: The method includes the following steps: Step 1. Using a meta-network to capture the rules of adaptive network traffic optimization tasks under different network environments and resource configurations; in order to capture the dynamics of network traffic at multiple time scales, a hierarchical temporal abstraction is established, and a meta-network is constructed based on an autoencoder to learn the rules of traffic at different scales, thereby enhancing the method's ability in long-term planning; Step 2. Introducing a context-aware meta-strategy generator to learn the mapping relationship between different scenario requirements and the trained meta-network, and to achieve rapid adaptation and adjustment to unknown scenarios to improve network performance; establishing an adaptive meta-learning framework to update the strategy; Step 3. Constructing collaborative optimization between multiple autonomous systems or networks in a federated autonomous learning architecture.
2. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 1, characterized in that: Using the Priority Experience Replay algorithm with replay cache, the meta-network and strategy table are implemented in the reinforcement learning framework to learn step 1 and the learning strategy; Meta-network is used to capture traffic patterns in different dynamic traffic scenarios to characterize the learned and trained network traffic QoS parameters. A multi-layer hierarchical timing abstraction mechanism is designed to calculate traffic indicators in a low aggregation time, thereby improving the ability of meta-network encoder and decoder to extract feature information representation of traffic.
3. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 2, characterized in that: The QoS configuration is obtained by weighted summation. ; in, Overall quality of service configuration, For the The weight of each QoS parameter, For the The value of QoS parameters; a single meta-network and a policy table are used to jointly model QoS parameters. The reinforcement learning environment first provides traffic data patterns or resource configurations as inputs for training the meta-network; in order to capture multi-scale dynamic traffic patterns, fine-grained and medium-scale QoS objectives are adopted; in step 2, the context-aware policy generator uses meta-policies to optimize the weight allocation of target QoS parameters; the context-aware policy network first collects QoS parameter data of traffic to represent the current state, then it generates a set of policy parameter-based weights in the best meta-network by querying the meta-network, and finally generates a set of optimized QoS weight parameters according to the given QoS parameters at multiple scales; the policy table is represented by a set of QoS hyperparameters, which assign different priorities to QoS parameters and learn QoS configuration schemes; ; in, For the The value of a QoS parameter, Strategy generation algorithm, For the indivual The hyperparameters of the parameters, The current network context; based on the policy generation algorithm, the QoS hyperparameters are adjusted to obtain the QoS parameter values; based on the QoS parameter values, the model generates traffic configuration according to actual needs; and the traffic configured in the policy table is implemented.
4. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 2, characterized in that: Based on the policy gradient optimization method, the meta-algorithm and the autoencoder are combined for meta-network training. In step 2, the meta-learning agent updates the policy table according to the environment interaction, and designs a scene-fixed accumulated experience replay storage buffer to collect experience data for training the meta-network. The meta-experience consists of a series of states, actions, and rewards of old scenes. By caching and replaying the data from the training meta-network, the difference between the distribution of sampled data and the target distribution is continuously reduced. Adapt to the old environment through the context-aware policy network, design a meta-experience replay storage buffer fixed to the scene accumulation, interact with the environment with different policy meta-agents; use transfer meta and policy meta-table to interact in the old environment.
5. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 1, characterized in that: A distributed collaborative learning framework is designed to encourage cooperation between multiple autonomous systems or network domains; A unified supervisory signal is generated based on the global QoS goal to guide the strategy adjustment. In order to avoid the additional overhead of frequent communication between networks, a method based on federated learning is adopted to mine the common information between multiple autonomous systems or multiple network domains. Each domain updates the strategy locally according to its global weight.
6. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 1, characterized in that: Each meta-network is updated and trained through the contextual attention mechanism. The meta-learning agent uses a shared meta-strategy to generate a combination of randomly sampled strategies when generating the state of the current scene to improve the generalization ability of the strategy. The contextual attention mechanism uses the following algorithm: a i ; a i For the The attention weight of each element, is the learnable parameter matrix, For the The hidden state of an element, is the query vector; Sample a set of different policies from all policy vectors, assign an independent attention mechanism weight to the additional spatial vectors in the state of the generated meta-learning agent, and obtain the policy corresponding to the current environment; The environmental state is generated by the network operation indicators composed of the bottleneck resources, network node bandwidth, delay, throughput and flow ratio of the current network flow; when the multi-objective optimization method is adopted, the meta-learning agent dynamically adjusts the learning rate of each meta-network based on the difference in the objective function value using game theory. The dynamic learning rate adjustment formula is as follows: ; in, For the The learning rate of the meta-network, is the basic learning rate, For the The value of the objective function, Controls the sensitivity of changes in the learning rate for the tuning coefficient.
7. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 6, characterized in that: The federated learning weight that should be assigned to each autonomous system or network domain is estimated through a contribution evaluation algorithm based on Shapley value, and common supervision signals are dynamically assigned to different autonomous systems in the federated learning framework to ensure that the objective function can converge. According to the Shapley value formula, each policy table will obtain a federated weight to ensure that the information distribution of all policy tables is correctly adjusted; the Shapley value evaluation determines the attention distribution of each network or autonomous system, which is used to share policy information to ensure that the policy training achieves good performance. The Shapley value calculation formula is: ; in, Player The Shapley value of is the set of all players, Does not include players A subset of Characteristic function, which represents the value of the alliance.
8. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 7, characterized in that: The policy table uses a logarithmic function to obtain the distribution; based on the Shapley value, the sharing ratio of the global supervision signal is: For each strategy, federated learning training round, network domain, and time period, Representative Moment Network domain in The Shapley value of federated learning training rounds; is the number of learning rounds; all policy tables receive the global supervision signal shared by the federation; then the policy parameters are regenerated, the target is updated, and the policy parameters are optimized. In the federated learning training mode, each domain updates the local policy according to its global weight. The policy update formula is: ,in, is the policy function, For the current strategy, the method is adjusted continuously over several cycles to obtain the global goal. After the formula is converted, it is obtained: , For the moment The strategy parameters, Learning Rate is the policy gradient, For the status Next select action The probability of For the state-action value function, gradient approximation is introduced to calculate it, and the gradient is obtained through the small batch strategy table; the training rounds of the network environment are traversed, and data is collected iteratively. When m is large enough, the gradient approximation calculation when each small batch is a single strategy table replaces the global gradient calculation.
9. The multi-objective reinforcement learning method for adaptive network traffic optimization according to claim 8, characterized in that: A knowledge distillation method is used to provide sharing of experience information between network areas or autonomous systems; each policy table is assigned a weight, and some policy tables are randomly selected from them to generate similar experience data in a given replay memory.
10. A system for executing the multi-objective reinforcement learning method for adaptive network traffic optimization according to any one of claims 1 to 9, characterized in that: include: Meta-network training module, which is used to use meta-network to capture the laws of adaptive network traffic optimization tasks under different network environments and resource configurations; in order to capture the dynamics of network traffic at multiple time scales, a meta-network is constructed to learn the laws of traffic at different scales and enhance the ability of the method in long-term planning; A context-aware meta-policy generator module, including a meta-policy network training module and a policy network update module; The meta-policy network training module includes: a context-aware policy generator to enhance the policy table for rapid adaptation to different network environments and user demand scenarios; The adaptive meta-learning module includes a context-aware policy generator and a policy table learning module for updating the policy table. Different learning rates are designed for different meta-tables during policy table training. Each policy table has the same impact on the overall objective function. The learning rate of each meta-table is calculated using game theory formulas based on the objective function value of the strategy. The federated learning module is used to build collaborative optimization between multiple autonomous systems or networks in a federated autonomous learning architecture, build a distributed collaborative learning framework, and realize collaborative learning of multi-feature network areas. A contribution algorithm based on Shapley value is introduced to dynamically adjust the weights of different network areas or autonomous systems in the learning process, and a knowledge distillation method is introduced to improve the mobility and adaptability of the network model.
Citation Information
Cited By
AI-driven real-time network optimization algorithm
CN120128494A
Tidal flow network optical path planning method, manager, controller and system based on meta-learning
CN120186501A
Tidal flow network optical path planning method and system based on meta learning
CN120186501B
Multi-working-condition process parameter optimization method and system based on migration reinforcement learning
CN120235054A
Intelligent scheduling and collaboration system and method for city reconstruction full life cycle
CN120672163A
