A communication network intelligent management system based on multi-agent reinforcement learning

Abstract

The application discloses a kind of communication network intelligent management systems based on multi-agent reinforcement learning, including communication protocol module, state perception module, data storage and analysis module, agent module, automation decision module and resource management module;Communication protocol module defines and manages the communication standard and rule between each agent according to network environment and performance requirement, state perception module is based on sensor real-time collection network equipment's network traffic data, data storage and analysis module processes network traffic data, agent module is based on the network traffic data after processing, using deep neural network and reinforcement learning model obtains optimal flow adjustment strategy, automation decision module generates network management decision, according to the decision, automatically adjusts network configuration, optimizes resource allocation and prevents potential failure, resource management module dynamically adjusts the configuration of network resources.The application reduces the dependence on artificial intervention, significantly improves the efficiency and response speed of network management.

Description

TECHNICAL FIELD

[0001] The application relates to the field of artificial intelligence and communication network technology, in particular to a communication network intelligent management system based on multi-agent reinforcement learning. BACKGROUND

[0002] With the development of automation and intelligent technology, network management is gradually transforming to automation operation to reduce the dependence on manual intervention, improve management efficiency and response speed, at the same time, the network security situation is becoming increasingly severe, the complexity of network attacks and security threats is increasing, and the network management system is required to have strong real-time monitoring and defense capabilities, the traditional network management method often relies on manual configuration and monitoring, which is difficult to adapt to the needs of large-scale, dynamic and complex network environment, especially in today's network traffic surge, various types of equipment and security threats, how to effectively manage the communication network to ensure the stability, security and efficiency of the network has become a problem to be solved, the existing technology still has many deficiencies in realizing the automation, intelligentization and security of network management, usually adopts a centralized management architecture, and a central controller is responsible for the management and decision of the whole network, however, this architecture has poor scalability and single point failure risk when facing large-scale networks, in addition, the traditional management method often relies on fixed rules and strategies, lacks self-adaptation and self-learning ability, and is difficult to respond to the changes of network state and the emergence of security threats. SUMMARY

[0003] The purpose of the application is to provide a communication network intelligent management system based on multi-agent reinforcement learning, which realizes the automation, intelligentization and security of communication network management through an automated and intelligent management method, and provides more efficient, safe and stable network services for users.

[0004] In order to solve the above technical problems, the application adopts the following technical scheme:

[0005] A communication network intelligent management system based on multi-agent reinforcement learning, comprising:

[0006] A communication protocol module, a state perception module, a data storage and analysis module, an agent module, an automated decision module and a resource management module.

[0007] The communication protocol module, the state perception module, the data storage and analysis module, the agent module, the automated decision module and the resource management module are electrically connected in sequence, and the resource management module is electrically connected with the agent module.

[0008] The communication protocol module is used for defining and managing the communication standards and rules between the agents according to the network environment and performance requirements, including identity authentication, synchronization mechanism and data encryption.

[0009] a state-aware module configured to collect network traffic data of the network device in real time based on sensors, the data including bandwidth utilization, latency, packet loss rate, device temperature, and power consumption, and perform anomaly detection on the network to timely discover potential problems or faults in the network and provide early warning information for the agent module.

[0010] a data storage and analysis module configured to clean, preprocess, and store the network traffic data in the state-aware module to obtain preprocessed network traffic data.

[0011] an agent module configured to obtain time features and optimal traffic adjustment strategies based on the preprocessed network traffic data using deep neural networks and reinforcement learning models, and update the strategies based on decision effect data.

[0012] an automated decision module configured to generate network management decisions based on the results of the agent module, and automatically adjust network configurations, optimize resource allocation, and prevent potential faults according to the decisions.

[0013] a resource management module configured to dynamically adjust the configurations of network resources, including bandwidth allocation, routing selection, and device scheduling, based on the results of the automated decision module and the agent module.

[0014] Further, the communication protocol module is configured to perform the following actions:

[0015] Step 1: Use security protocol technology to authenticate the valid certificate or key provided by the agent, and perform information interaction after successful authentication.

[0016] Step 2: Through information interaction, establish a synchronization mechanism to obtain and parse data packets between agents, the data packets including timestamps and version information, the timestamps being used to identify the sending time points of the data packets, and the version information being used to indicate the update status and version number of the data, sort and integrate the data information, synchronize the timing and state between the agents, and encrypt the data packets.

[0017] Step 3: Use CRC (Cyclic Redundancy Check) technology to check the integrity of the data packets, if the check result is normal, do not perform any operation, otherwise send a data packet retransmission instruction until the check result is normal.

[0018] The check includes the following contents:

[0019] The CRC register is initialized to 0, each bit of the data is XORed with the current value of the CRC register, and the result is left shifted by one bit, if the highest bit after left shift is 1, the value is XORed with the polynomial to obtain the final CRC check code, and the final CRC check code is appended to the end of the original data and sent to the receiving end; the CRC value of the original data and the CRC value of the data received by the receiving end are calculated, if they are the same, the check is passed, otherwise the check is failed.

[0020] The calculation formula of the CRC value is:

[0021]

[0022] Wherein, M(x) represents the original information polynomial, x represents the formal variable, n represents the total number of bits of the CRC check code, k represents the number of bits of the original data, G(x) represents the generating polynomial, Q(x) represents the quotient of two polynomials, and R(x) represents the remainder polynomial.

[0023] Further, the data storage and analysis module is configured to perform the following actions:

[0024] The network traffic data in the state perception module is cleaned to remove noise and outliers, and the cleaned data is converted into time series data. The cleaned data is compressed, normalized and feature extracted to obtain preprocessed network traffic data.

[0025] Further, the intelligent agent module is configured to perform the following actions:

[0026] Step 1, based on a deep neural network, a traffic pattern analysis model is constructed, the preprocessed data is input into the model, and time features are extracted; the traffic pattern analysis model is trained using historical network traffic data, and the mean square error is used as a loss function to measure the prediction error; the preprocessed data is input into the trained traffic pattern analysis model to obtain the predicted traffic peak.

[0027] Step 2, based on real-time state and historical experience, the optimal action is selected using a reinforcement learning model to adjust the traffic allocation rules.

[0028] Step 3, decision effect data is collected through an event-based triggering mechanism, and the strategy is updated according to the feedback results; wherein, the events include time periodic events, traffic threshold events, performance index abnormal events and external triggering events, the decision effect data includes the network traffic before and after the decision execution, the network performance index and the perception and feedback of the user's network performance change; the decision effect is evaluated by comparing the performance index before and after the decision execution and analyzing the change of traffic pattern.

[0029] Further, in step 2, adjusting the traffic distribution rule includes the following sub-steps:

[0030] Step 201, abstract the communication network into a reinforcement learning environment, wherein the real-time state is composed of network state and time characteristics, and the reward function is a measure of optimizing network performance;

[0031] Step 202, initialize network parameters, and set up two identical network structures, one for online prediction and the other for stable target values; input the network state into the reinforcement learning model to output the expected Q value of each action; execute the traffic adjustment strategy, observe the state transition and the reward obtained, and store them into the experience replay pool; randomly draw a batch of experiences from the experience replay pool to train the reinforcement learning model; for each batch of experiences, use the trained reinforcement learning model to predict the Q value of the current state, and use the target network to predict the Q value of the next state to calculate the target Q value; update the parameters of the trained reinforcement learning model by minimizing the error between the predicted Q value and the target Q value, and periodically copy the parameters of the trained reinforcement learning model to the target network; according to the current network state and the time characteristics extracted in step 1, predict the Q value of each action through the trained reinforcement learning model, and select the action with the highest Q value as the optimal traffic adjustment strategy.

[0032] Further, in step 3, the calculation formula of decision effect evaluation is:

[0033] The formula for calculating performance difference is:

[0034] text difference value = decision after index value - decision before index value

[0035] The formula for calculating performance improvement percentage is:

[0036]

[0037] The formula for calculating performance deterioration percentage is:

[0038]

[0039] The formula for calculating user satisfaction score is:

[0040]

[0041] Wherein, A represents the total number of users participating in feedback.

[0042] Compared with the prior art, the beneficial effects of the present application are:

[0043] The communication network intelligent management system provided by the application reduces the dependence on manual intervention through an automated and intelligent management mode, significantly improves the efficiency and response speed of network management, especially in the face of large-scale, dynamic and complex network environments, can quickly make decisions and adjustments to adapt to changes in network status, realizes the automation, intelligence and security of communication network management, significantly improves the efficiency and response speed of network management, enhances the security and stability of the network, optimizes resource allocation and traffic management, and provides users with more efficient, safe and stable network services. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The system structure framework of the application is shown in the figure. DETAILED DESCRIPTION

[0045] The embodiments of the application will be described in detail below, examples of which are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the application, and cannot be interpreted as a limitation on the application.

[0046] To achieve the above-mentioned purpose, the application provides a communication network intelligent management system based on multi-agent reinforcement learning, as shown in Figure 1 The system comprises:

[0047] A communication protocol module, a state perception module, a data storage and analysis module, an agent module, an automated decision module and a resource management module.

[0048] The communication protocol module, the state perception module, the data storage and analysis module, the agent module, the automated decision module and the resource management module are electrically connected in sequence, and the resource management module is electrically connected with the agent module.

[0049] The communication protocol module serves as the cornerstone of the entire system, and is used to define and manage the communication standards and rules between the agents in the system according to the network environment and performance requirements, including identity verification, synchronization mechanism and data encryption, to ensure that the agents can communicate seamlessly, these protocols not only need to support real-time data exchange, but also need to consider the heterogeneity, scalability and security of the network, through defining a unified message format, transmission protocol and encryption mechanism, the module ensures the consistency and integrity of the data, and provides a solid communication foundation for the collaborative work between the agents.

[0050] The specific content is:

[0051] Step 1, using security protocol technology, identity verification is performed on the valid certificate or key provided by the agent, after successful verification, a secure communication session is established between the agents for information exchange.

[0052] Step 2: Through information interaction, establish a synchronization mechanism to obtain data packets between agents and perform analysis. The data packet includes a timestamp and version information. The timestamp is used to identify the sending time point of the data packet, which ensures that all agents can understand the sending order of the data, thereby avoiding timing confusion. The version information is used to indicate the update status and version number of the data, which helps the receiving end to determine whether the received data is the latest and whether subsequent processing based on the new version of the data is needed. State synchronization is achieved by exchanging state information packets periodically or on demand. The information packet includes the current state of the agent, resource occupation, and decision result, so that all agents can maintain a common understanding of the network environment and consistent action. Sort and integrate the data information. Sorting ensures that the data is processed in the order of the sending time point, thereby avoiding incorrect decisions due to timing confusion. Integration is to combine related data sent by multiple agents into a unified data view for subsequent analysis and decision-making. Synchronize the timing and state between agents.

[0053] To ensure the confidentiality and security of data transmission, symmetric key encryption algorithm or asymmetric key encryption algorithm is used to encrypt the data packet, ensuring that even if the data is intercepted during transmission, it cannot be easily decrypted by unauthorized third parties. At the same time, the communication protocol also specifies the management and update mechanism of the key to ensure the security and timeliness of the key.

[0054] Step 3: To verify the integrity of the transmitted data, CRC (Cyclic Redundancy Check) technology is used to check the integrity of the data packet. If the check result is normal, no operation is performed, otherwise a retransmission instruction is sent until the check result is normal.

[0055] The check includes the following contents:

[0056] Initialize the CRC register to 0, XOR each bit of the data with the current value of the CRC register, and left shift the result by one bit. If the highest bit after left shift is 1, XOR the value with the polynomial to get the final CRC check code. Attach the final CRC check code to the end of the original data and send it to the receiving end. Calculate the CRC value of the original data and the CRC value of the received data at the receiving end. If they are the same, the check is passed, indicating that the data has not been tampered with or errors have occurred during transmission. The receiving end will process the data normally, otherwise it indicates that there is a problem with the data during transmission, and the check is not passed.

[0057] The calculation formula of the CRC value is:

[0058]

[0059] Wherein, M(x) represents the original information polynomial, x represents the formal variable, n represents the total number of CRC check code bits, k represents the number of bits of the original data, G(x) represents the generating polynomial, Q(x) represents the quotient of two polynomials, and R(x) represents the remainder polynomial.

[0060] The state-aware module is used to collect network traffic data of the network device in real time based on sensors, and the data includes bandwidth utilization, delay, packet loss rate, device temperature and power consumption. After preliminary screening and filtering, the data is transmitted to the data storage and analysis module. In addition, the module has abnormal detection capability and can timely discover potential problems or faults in the network to provide early warning information for the agent module.

[0061] In this embodiment, high-precision hardware sensors and SNMP agents are deployed at key nodes of the network, such as key ports of Cisco routers and Juniper switches, to capture traffic data and device status in real time.

[0062] The data storage and analysis module is the center of data processing, which is used to clean, preprocess and store the data in the state-aware module, and use advanced data analysis techniques to mine the rules and trends behind the data to provide decision support for the agent module. The specific content is:

[0063] The network traffic data in the state-aware module is cleaned to remove noise and outliers, and the cleaned data is converted into time series data. The cleaned data is compressed, normalized and feature extracted to obtain preprocessed network traffic data.

[0064] In this embodiment, for high-traffic Cisco router interfaces, the strategy may include collecting bandwidth usage and packet loss rate once per second, while for relatively static configuration information, it may only need to collect once per minute or hour.

[0065] The agent module is the core of the entire system, which is used to use deep neural networks and reinforcement learning models to gradually adapt to changes in the network environment by constantly trying and learning based on preprocessed network traffic data, obtain the optimal traffic adjustment strategy, and update the strategy based on decision effect data to maximize network performance. In addition, agents also form alliances or competitive relationships to jointly cope with complex network environments. The specific content is:

[0066] Step 1, analyze network traffic patterns and predict traffic peaks using convolutional neural networks. The specific content is:

[0067] Based on the convolutional neural network, a traffic pattern analysis model is constructed, and the preprocessed data is input into the model to extract time features.

[0068] The traffic pattern analysis model is trained using historical network traffic data, and the mean square error is used as a loss function to measure the prediction error.

[0069] The preprocessed data is input into the trained traffic pattern analysis model to obtain the predicted traffic peak.

[0070] Step 2, based on real-time state and historical experience, use DQN (Deep Q-Network) model to select the optimal action and adjust the traffic allocation rule. The specific content is:

[0071] Step 201, abstract the communication network into a reinforcement learning environment, where the real-time state is composed of network state and time characteristics, and the reward function is designed to optimize the network performance metrics.

[0072] Step 202, initialize network parameters and set up two identical network structures, one for online prediction and the other for stable target value; use the DQN model to input the state as input and output the expected Q value of each possible action; execute the traffic adjustment strategy in the communication network environment, observe the state transition and the reward obtained, and store them in the experience replay pool; randomly draw a batch of experiences from the experience replay pool to train the DQN model; for each batch of experiences, use the main network to predict the Q value of the current state, and use the target network to predict the Q value of the next state to calculate the target Q value; update the parameters of the main network by minimizing the error between the predicted Q value and the target Q value, and periodically copy the parameters of the main network to the target network; according to the current network state and the time characteristics extracted in step 1, predict the Q value of each action through the main network, and select the action with the highest Q value as the optimal traffic adjustment strategy.

[0073] Step 3, collect decision effect data through event-based triggering mechanism, and update the strategy according to the feedback results. The specific content is:

[0074] Events include time periodic events, traffic threshold events, performance indicator abnormal events and external trigger events, decision effect data includes network traffic before and after decision execution, network performance indicators and user's perception and feedback of network performance changes; by comparing the performance indicators before and after decision execution and analyzing whether the change of traffic pattern meets the expectation, the decision effect is evaluated.

[0075] The calculation formula of decision effect evaluation is:

[0076] Performance difference calculation:

[0077] text difference value = decision after indicator value - decision before indicator value

[0078] Performance improvement percentage:

[0079]

[0080] Performance deterioration percentage:

[0081]

[0082] User satisfaction score:

[0083]

[0084] Wherein, A represents the total number of users participating in feedback.

[0085] The automatic decision module is used to generate network management decisions based on the results of the agent module, and automatically adjust network configurations, optimize resource allocation, and prevent potential failures according to the decisions. The decision-making process follows preset optimization objectives and constraints to ensure the rationality and effectiveness of the decisions. At the same time, the module also has adaptive ability to adjust the decision-making strategy according to actual feedback, realizing the continuous optimization of the network.

[0086] The resource management module is the executor of dynamic adjustment of system resources, which is used to dynamically adjust the configuration of network resources, including bandwidth allocation, routing selection and device scheduling, according to the results of the automatic decision module and the agent module. Through accurate control of resource allocation and use, the resource management module can ensure the effective use of network resources, improve the overall performance and stability of the network, and at the same time, the module also has load balancing and fault recovery capability, which can quickly adjust the strategy when the network appears fluctuation or failure, to guarantee the continuity and reliability of the network.

[0087] In summary, the present application significantly improves the network management efficiency and response speed through automation and intelligent technology, especially in complex dynamic network environment, the system enhances the network security, has adaptive learning and optimization ability, improves the resource utilization, realizes intelligent traffic management and prediction, overall, the system improves the stability, security and user experience of the network, promotes the intelligent process of communication network management.

[0088] Although the embodiments of the present application are shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application. Any various other corresponding changes and variations made according to the technical concept of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A multi-agent reinforcement learning based intelligent management system for communication networks, characterized in that, The application relates to a network management system based on multi-agent technology, which comprises a communication protocol module, a state sensing module, a data storage and analysis module, an agent module, an automated decision module and a resource management module. The communication protocol module, the state sensing module, the data storage and analysis module, the agent module, the automated decision module and the resource management module are sequentially electrically connected, and the resource management module is electrically connected with the agent module. The communication protocol module is used for defining and managing the communication standards and rules between the agents according to the network environment and performance requirements, including identity authentication, synchronization mechanism and data encryption. The state sensing module is used for collecting the network traffic data of the network equipment in real time based on sensors, the data including bandwidth utilization, delay, packet loss rate, equipment temperature and power consumption, and performing abnormal detection on the network to provide early warning information. The data storage and analysis module is used for cleaning, preprocessing and storing the network traffic data in the state sensing module to obtain the preprocessed network traffic data. The agent module is used for obtaining time characteristics and optimal traffic adjustment strategies based on the preprocessed network traffic data by using a deep neural network and a reinforcement learning model, and updating the strategies based on decision effect data. The automated decision module is used for generating network management decisions based on the results of the agent module, and automatically adjusting network configurations, optimizing resource allocation and preventing potential faults according to the decisions. The resource management module is used for dynamically adjusting the configuration of network resources, including bandwidth allocation, routing selection and equipment scheduling, according to the results of the automated decision module and the agent module. The communication protocol module is configured to perform the following actions:

2. The multi-agent reinforcement learning based intelligent management system for communication network according to claim 1, wherein, Step 1, identity authentication of the certificate or key provided by the agent is performed by using a security protocol technology, and information interaction is performed after successful authentication; Step 2, a synchronization mechanism is established through information interaction to obtain and analyze data packets between the agents, the data packets including a timestamp and version information, the timestamp being used for identifying the sending time point of the data packet, the version information being used for indicating the update state and version number of the data, the data information being sorted and integrated, the timing and state between the agents being synchronized, and the data packet being encrypted; Step 3, a cyclic redundancy check technology is adopted to check the integrity of the data packet, if the check result feedback is normal, no operation is performed, otherwise a data packet retransmission instruction is sent until the check result feedback is normal. In step 3, the check 3. The multi-agent reinforcement learning based intelligent management system for communication network according to claim 2, characterized in that, includes the following contents: the cyclic redundancy check register is initialized to 0, each bit of the data is XORed with the current value of the cyclic redundancy check register, and the result is left shifted by one bit, if the highest bit after left shift is 1, the value is XORed with a polynomial to obtain the final cyclic redundancy check code, the final cyclic redundancy check code is appended to the end of the original data and sent to the receiving end; the cyclic redundancy check value of the original data and the cyclic redundancy check value of the data received by the receiving end are calculated, if both are the same, the check is passed, otherwise the check is failed; the calculation formula of the CRC value is: ​ Wherein, M(x) represents the original information polynomial, x represents the formal variable, n represents the total number of cyclic redundancy check check code, k represents the bit number of the original data, G(x) represents the generating polynomial, Q(x) represents the quotient of two polynomials, and R(x) represents the remainder polynomial.

4. The multi-agent reinforcement learning based intelligent management system for communication network according to claim 1, wherein, The data storage and analysis module is configured to perform the following actions: The network traffic data in the state perception module is cleaned to remove noise and outliers, and the cleaned data is converted into time series data; the cleaned data is compressed, normalized and feature extracted to obtain preprocessed network traffic data. 5.The multi-agent reinforcement learning based intelligent management system for communication network according to claim 1, wherein, The agent module is configured to perform the following actions: Step 1, based on a deep neural network, a traffic pattern analysis model is constructed, and the preprocessed data is input into the model to extract time features; The traffic pattern analysis model is trained using historical network traffic data, and the mean square error is used as the loss function to measure the prediction error; The preprocessed data is input into the trained traffic pattern analysis model to obtain the predicted traffic peak; Step 2, based on real-time state and historical experience, an optimal action is selected using a reinforcement learning model to adjust the traffic allocation rule; Step 3, collect decision effect data through an event-based triggering mechanism, and update the strategy according to the feedback results; wherein, the events include time periodic events, traffic threshold events, performance index abnormal events and external triggering events, and the decision effect data includes the network traffic before and after the decision execution, the network performance index and the perception and feedback of the user's network performance change; The decision effect is evaluated by comparing the performance index before and after the decision execution and analyzing the change of the traffic pattern.

6. The multi-agent reinforcement learning based intelligent management system for communication network according to claim 5, wherein, In step 2, adjusting the traffic allocation rule includes the following sub-steps: Step 201, abstract the communication network into a reinforcement learning environment, wherein the real-time state is composed of network state and time feature, and the reward function is the measurement of optimizing network performance; Step 202, initialize network parameters and set two identical network structures, one for online prediction and the other for stable target value; input the network state into the reinforcement learning model to output the expected Q value of each action; Execute the traffic adjustment strategy, observe the state transition and the reward obtained, and store them in the experience replay pool; Randomly draw a batch of experiences from the experience replay pool to train the reinforcement learning model; for each batch of experiences, use the trained reinforcement learning model to predict the Q value of the current state, and use the target network to predict the Q value of the next state to calculate the target Q value; Update the parameters of the trained reinforcement learning model by minimizing the error between the predicted Q value and the target Q value, and periodically copy the parameters of the trained reinforcement learning model to the target network; according to the current network state and the time feature extracted in step 1, predict the Q value of each action through the trained reinforcement learning model, and select the action with the highest Q value as the optimal traffic adjustment strategy.

7. The multi-agent reinforcement learning based intelligent management system for communication network according to claim 5, wherein, In step 3, the calculation formula of decision effect evaluation is: The formula for calculating the performance difference is: text difference value = decision after index value - decision before index value The formula for calculating the performance improvement percentage is: The formula for calculating the performance deterioration percentage is: The formula for calculating the user satisfaction score is: Where A represents the total number of users participating in feedback.

Images