Communication network intelligent management system based on multi-agent reinforcement learning

Abstract

The invention discloses a communication network intelligent management system based on multi-agent reinforcement learning. The system comprises a communication protocol module, a state sensing module, a data storage and analysis module, an agent module, an automatic decision module and a resource management module. The communication protocol module defines and manages communication standards and rules among the intelligent agents according to a network environment and performance requirements, the state sensing module collects network flow data of network equipment in real time based on a sensor, the data storage and analysis module processes the network flow data, and the intelligent agent module sends the network flow data to the intelligent agent module based on the processed network flow data. An optimal flow adjustment strategy is obtained by using a deep neural network and a reinforcement learning model, an automatic decision-making module generates a network management decision, network configuration is automatically adjusted according to the decision, resource allocation is optimized and potential faults are prevented, and a resource management module dynamically adjusts configuration of network resources. According to the invention, the dependence on manual intervention is reduced, and the efficiency and response speed of network management are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and communication network technology, and in particular to a communication network intelligent management system based on multi-agent reinforcement learning. Background Art

[0002] With the development of automation and intelligent technology, network management is gradually transforming to automated operation and maintenance to reduce dependence on manual intervention and improve management efficiency and response speed. At the same time, the network security situation is becoming increasingly severe, and the complexity of network attacks and security threats is increasing, requiring network management systems to have strong real-time monitoring and defense capabilities. Traditional network management methods often rely on manual configuration and monitoring, which is difficult to adapt to the needs of large-scale, dynamic and complex network environments. Especially in today's world where network traffic is surging, equipment types are diverse, and security threats are emerging in an endless stream, how to effectively manage communication networks and ensure network stability, security and efficiency has become an urgent problem to be solved. Existing technologies still have many shortcomings in realizing the automation, intelligence and security of network management. They usually adopt a centralized management architecture, with the central controller responsible for the management and decision-making of the entire network. However, this architecture has problems such as poor scalability and high risk of single point failure when facing large-scale networks. In addition, traditional management methods often rely on fixed rules and strategies, lack the ability to adapt and self-learn, and are difficult to respond to changes in network status and the emergence of security threats in a timely manner. Summary of the invention

[0003] The purpose of the present invention is to provide a communication network intelligent management system based on multi-agent reinforcement learning, which realizes the automation, intelligence and security of communication network management through automated and intelligent management methods, and provides users with more efficient, secure and stable network services.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A communication network intelligent management system based on multi-agent reinforcement learning, comprising:

[0006] Communication protocol module, state perception module, data storage and analysis module, intelligent agent module, automated decision-making module and resource management module.

[0007] The communication protocol module, the state perception module, the data storage and analysis module, the intelligent body module, the automatic decision module and the resource management module are electrically connected in sequence, and the resource management module is electrically connected to the intelligent body module.

[0008] The communication protocol module is used to define and manage the communication standards and rules between various intelligent agents according to the network environment and performance requirements, including: identity authentication, synchronization mechanism and data encryption.

[0009] The state perception module is used to collect network traffic data of network devices in real time based on sensors. The data includes bandwidth utilization, delay, packet loss rate, device temperature and power consumption, and perform network anomaly detection. It can promptly discover potential problems or failures in the network and provide early warning information for the intelligent body module.

[0010] The data storage and analysis module is used to clean, preprocess and store the network flow data in the state perception module to obtain the preprocessed network flow data.

[0011] The intelligent agent module is used to obtain time characteristics and optimal traffic adjustment strategies based on preprocessed network traffic data using deep neural networks and reinforcement learning models, and update the strategy based on decision-making effect data.

[0012] The automated decision-making module is used to generate network management decisions based on the results of the intelligent agent module, and automatically adjust the network configuration, optimize resource allocation and prevent potential failures according to the decisions.

[0013] The resource management module is used to dynamically adjust the configuration of network resources, including bandwidth allocation, route selection and device scheduling, based on the results of the automated decision-making module and the intelligent agent module.

[0014] Furthermore, the communication protocol module is configured to perform the following actions:

[0015] Step 1: Use security protocol technology to authenticate the valid certificate or key provided by the intelligent agent, and exchange information after successful verification.

[0016] Step 2: Through information interaction, establish a synchronization mechanism to obtain and parse the data packets between intelligent agents. The data packets include timestamps and version information. The timestamp is used to identify the time point when the data packet is sent, and the version information is used to indicate the update status and version number of the data. The data information is sorted and integrated, the timing and status between intelligent agents are synchronized, and the data packets are encrypted.

[0017] Step 3: Use CRC (Cyclic Redundancy Check) technology to check the integrity of the data packet. If the check result feedback is normal, no operation is performed. Otherwise, a data packet retransmission instruction is sent until the check result feedback is normal.

[0018] The verification includes the following:

[0019] Initialize the CRC register to 0, perform an XOR operation on each bit of the data and the current value of the CRC register, and shift the result left by one bit. If the highest bit after the left shift is 1, perform an XOR operation on the value and the polynomial to obtain the final CRC check code. The final CRC check code is attached to the end of the original data and sent to the receiving end. Calculate the CRC value of the original data and the CRC value of the data received by the receiving end. If the two are the same, the check passes, otherwise the check fails.

[0020] The calculation formula of CRC value is:

[0021]

[0022] Among them, M(x) represents the original information polynomial, x represents the formal variable, n represents the total number of bits of the CRC check code, k represents the number of bits of the original data, G(x) represents the generating polynomial, Q(x) represents the quotient of two polynomials, and R(x) represents the remainder polynomial.

[0023] Furthermore, the data storage and analysis module is configured to perform the following actions:

[0024] The network traffic data in the state perception module is cleaned to remove noise and outliers, and the cleaned data is converted into time series data. The cleaned data is compressed, normalized and feature extracted to obtain preprocessed network traffic data.

[0025] Furthermore, the agent module is configured to perform the following actions:

[0026] Step 1: Build a traffic pattern analysis model based on a deep neural network, input the preprocessed data into the model, and extract time features; use historical network traffic data to train the traffic pattern analysis model, and use the mean square error as the loss function to measure the prediction error; input the preprocessed data into the trained traffic pattern analysis model to obtain the predicted traffic peak.

[0027] Step 2: Based on real-time status and historical experience, use the reinforcement learning model to select the optimal action and adjust the traffic distribution rules.

[0028] Step 3: Collect decision effect data through an event-based trigger mechanism and update the strategy based on the feedback results; events include time periodic events, traffic threshold events, performance indicator abnormal events and external trigger events; decision effect data includes network traffic conditions before and after decision execution, network performance indicators and user perception and feedback of network performance changes; evaluate the decision effect by comparing performance indicators before and after decision execution and analyzing changes in traffic patterns.

[0029] Furthermore, in step 2, adjusting the traffic distribution rule includes the following sub-steps:

[0030] Step 201, abstracting the communication network into a reinforcement learning environment, where the real-time state consists of the network state and time characteristics, and the reward function is a measure of optimizing network performance;

[0031] Step 202, initialize network parameters, and set two identical network structures, one for online prediction and the other for stabilizing the target value; input the network state into the reinforcement learning model, and output the expected Q value of each action; execute the traffic adjustment strategy, observe the state transition and the reward obtained, and store them in the experience replay pool; randomly extract a batch of experience from the experience replay pool to train the reinforcement learning model; for each batch of experience, use the trained reinforcement learning model to predict the Q value of the current state, and use the target network to predict the Q value of the next state to calculate the target Q value; update the parameters of the trained reinforcement learning model by minimizing the error between the predicted Q value and the target Q value, and regularly copy the parameters of the trained reinforcement learning model to the target network; predict the Q value of each action through the trained reinforcement learning model according to the current network state and the time features extracted in step 1, and select the action with the highest Q value as the optimal traffic adjustment strategy.

[0032] Furthermore, in step 3, the calculation formula for decision effect evaluation is:

[0033] The formula for calculating the performance difference is:

[0034] text difference value = index value after decision-index value before decision

[0035] The formula for calculating the performance improvement percentage is:

[0036]

[0037] The formula for calculating the performance degradation percentage is:

[0038]

[0039] The formula for calculating the user satisfaction score is:

[0040]

[0041] Among them, A represents the total number of users participating in the feedback.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The communication network intelligent management system proposed in the present invention reduces dependence on manual intervention through automated and intelligent management methods, significantly improving the efficiency and response speed of network management. In particular, when facing large-scale, dynamic and complex network environments, it can make decisions and adjustments quickly to adapt to changes in network status, thereby realizing automation, intelligence and security of communication network management, significantly improving the efficiency and response speed of network management, enhancing the security and stability of the network, and optimizing resource allocation and traffic management, providing users with more efficient, secure and stable network services. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a system structure framework diagram of the present invention. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0046] To achieve the above objectives, the present invention proposes a communication network intelligent management system based on multi-agent reinforcement learning, such as Figure 1 As shown, including:

[0047] Communication protocol module, state perception module, data storage and analysis module, intelligent agent module, automated decision-making module and resource management module.

[0048] The communication protocol module, the state perception module, the data storage and analysis module, the intelligent body module, the automatic decision module and the resource management module are electrically connected in sequence, and the resource management module is electrically connected to the intelligent body module.

[0049] Among them, the communication protocol module, as the cornerstone of the entire system, is used to define and manage the communication standards and rules between various intelligent agents in the system according to the network environment and performance requirements, including identity authentication, synchronization mechanism and data encryption, to ensure seamless communication between various intelligent agents. These protocols not only need to support real-time data exchange, but also need to consider the heterogeneity, scalability and security of the network. By defining a unified message format, transmission protocol and encryption mechanism, this module ensures the consistency and integrity of the data, providing a solid communication foundation for the collaborative work between intelligent agents.

[0050] The specific contents are:

[0051] Step 1: Use security protocol technology to authenticate the valid certificate or key provided by the intelligent agent. After successful verification, a secure communication session is established between the intelligent agents to exchange information.

[0052] Step 2: Establish a synchronization mechanism through information interaction to obtain and parse the data packets between agents. The data packets include timestamps and version information. The timestamp is used to identify the time point of sending the data packet. It ensures that all agents can understand the order of data sending, thereby avoiding timing confusion. The version information is used to indicate the update status and version number of the data, which helps the receiving end to determine whether the received data is the latest and whether subsequent processing based on the new version of the data is required. Status synchronization is achieved by exchanging status information packets regularly or on demand. The information packets include the current status of the agent, resource occupancy, and decision results, so that all agents can maintain a common understanding of the network environment and act in unison. The data information is sorted and integrated. Sorting ensures that the data is processed in the order of the sending time point, thereby avoiding erroneous decisions caused by timing confusion. Integration is to merge the relevant data sent by multiple agents into a unified data view for subsequent analysis and decision-making, and synchronize the timing and status between agents.

[0053] In order to ensure confidentiality and security during data transmission, symmetric key encryption algorithm or asymmetric key encryption algorithm is used to encrypt the data packet to ensure that even if the data is intercepted during transmission, it cannot be easily decrypted by unauthorized third parties. At the same time, the communication protocol also stipulates the key management and update mechanism to ensure the security and timeliness of the key.

[0054] Step 3: In order to verify the integrity of the transmitted data, the CRC (Cyclic Redundancy Check) technology is used to check the integrity of the data packet. If the check result feedback is normal, no operation is performed. Otherwise, a retransmission instruction is sent until the check result feedback is normal.

[0055] The verification includes the following:

[0056] Initialize the CRC register to 0, perform an XOR operation on each bit of data and the current value of the CRC register, and shift the result left by one bit. If the highest bit after the left shift is 1, perform an XOR operation on this value and the polynomial to obtain the final CRC check code, and append the final CRC check code to the end of the original data and send it to the receiving end; calculate the CRC value of the original data and the CRC value of the data received by the receiving end. If the two are the same, the check passes, indicating that there is no error or tampering in the data during transmission, and the receiving end will process the data normally. Otherwise, it indicates that there is a problem in the data transmission process and the check fails.

[0057] The calculation formula of CRC value is:

[0058]

[0059] Among them, M(x) represents the original information polynomial, x represents the formal variable, n represents the total number of bits of the CRC check code, k represents the number of bits of the original data, G(x) represents the generating polynomial, Q(x) represents the quotient of two polynomials, and R(x) represents the remainder polynomial.

[0060] The state perception module is used to collect network traffic data of network devices in real time based on sensors. The data includes bandwidth utilization, delay, packet loss rate, device temperature and power consumption. After preliminary screening and filtering, these data are passed to the data storage and analysis module. In addition, this module has the ability to detect anomalies, which can promptly detect potential problems or failures in the network and provide early warning information for the intelligent agent module.

[0061] In this embodiment, high-precision hardware sensors and SNMP agents are deployed at key nodes of the network, such as key ports of Cisco routers and Juniper switches, to capture traffic data and device status in real time.

[0062] The data storage and analysis module is the center of data processing, which is used to clean, preprocess and store the data in the state perception module, and use advanced data analysis technology to explore the laws and trends behind the data to provide decision support for the intelligent module. The specific contents are:

[0063] The network traffic data in the state perception module is cleaned to remove noise and outliers, and the cleaned data is converted into time series data. The cleaned data is compressed, normalized and feature extracted to obtain preprocessed network traffic data.

[0064] In this embodiment, for a high-traffic Cisco router interface, the policy may include collecting bandwidth usage and packet loss rate once per second, while for relatively static configuration information, it may only need to be collected once per minute or hour.

[0065] The intelligent agent module is the core of the entire system. It is used to gradually adapt to changes in the network environment based on pre-processed network traffic data, using deep neural networks and reinforcement learning models through continuous trial and error and learning, obtain the optimal traffic adjustment strategy, and update the strategy based on decision-making effect data to maximize network performance. In addition, the intelligent agents also form alliances or competitive relationships to jointly cope with complex network environments. The specific contents are:

[0066] Step 1: Analyze network traffic patterns using convolutional neural networks to predict traffic peaks. The specific content is:

[0067] A traffic pattern analysis model is built based on a convolutional neural network, and the preprocessed data is input into the model to extract time features.

[0068] The traffic pattern analysis model is trained using historical network traffic data, and the prediction error is measured using the mean square error as the loss function.

[0069] The preprocessed data is input into the trained traffic pattern analysis model to obtain the predicted traffic peak.

[0070] Step 2: Based on the real-time status and historical experience, use the DQN (Deep Q-Network) model to select the optimal action and adjust the traffic distribution rules. The specific contents are:

[0071] Step 201: abstract the communication network into a reinforcement learning environment, where the real-time state consists of the network state and time characteristics, and the reward function is designed to optimize the metric of network performance.

[0072] Step 202, initialize network parameters, and set two identical network structures, one for online prediction and the other for stabilizing the target value; use the state as input through the DQN model, and output the expected Q value of each possible action; execute the traffic adjustment strategy in the communication network environment, observe the state transition and the reward obtained, and store them in the experience replay pool; randomly extract a batch of experience from the experience replay pool to train the DQN model; for each batch of experience, use the main network to predict the Q value of the current state, and use the target network to predict the Q value of the next state to calculate the target Q value; update the parameters of the main network by minimizing the error between the predicted Q value and the target Q value, and periodically copy the parameters of the main network to the target network; predict the Q value of each action through the main network according to the current network state and the time features extracted in step 1, and select the action with the highest Q value as the optimal traffic adjustment strategy.

[0073] Step 3: Collect decision-making effect data through an event-based trigger mechanism and update the strategy based on the feedback results. The specific content is:

[0074] Events include time-periodic events, traffic threshold events, performance indicator abnormal events and external trigger events. Decision effect data include network traffic conditions before and after decision execution, network performance indicators and users' perception and feedback of network performance changes. The decision effect is evaluated by comparing performance indicators before and after decision execution and analyzing whether changes in traffic patterns are in line with expectations.

[0075] The calculation formula for decision effect evaluation is:

[0076] Performance difference calculation:

[0077] text difference value = index value after decision-index value before decision

[0078] Performance improvement percentage:

[0079]

[0080] Performance degradation percentage:

[0081]

[0082] User satisfaction rating:

[0083]

[0084] Among them, A represents the total number of users participating in the feedback.

[0085] The automated decision-making module is used to generate network management decisions based on the results of the intelligent agent module, and automatically adjust network configuration, optimize resource allocation, and prevent potential failures based on the decisions. The decision-making process follows the preset optimization goals and constraints to ensure the rationality and effectiveness of the decision. At the same time, the module also has adaptive capabilities and can adjust decision-making strategies based on actual feedback to achieve continuous optimization of the network.

[0086] The resource management module is the executor of the dynamic adjustment of system resources. It is used to dynamically adjust the configuration of network resources, including bandwidth allocation, routing selection, and device scheduling, based on the results of the automated decision-making module and the intelligent agent module. By precisely controlling the allocation and use of resources, the resource management module can ensure the effective use of network resources and improve the overall performance and stability of the network. At the same time, the module also has load balancing and fault recovery capabilities, and can quickly adjust strategies when network fluctuations or failures occur, ensuring the continuity and reliability of the network.

[0087] To sum up, the present invention significantly improves network management efficiency and response speed through automation and intelligent technology. Especially in complex and dynamic network environments, the system enhances network security, has adaptive learning and optimization capabilities, improves resource utilization, and realizes intelligent traffic management and prediction. Overall, the system improves network stability, security and user experience, and promotes the intelligent process of communication network management.

[0088] Although the embodiments of the present invention are shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present invention. Any other corresponding changes and modifications made according to the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A communication network intelligent management system based on multi-agent reinforcement learning, characterized in that: include: Communication protocol module, state perception module, data storage and analysis module, intelligent agent module, automated decision-making module and resource management module; The communication protocol module, the state perception module, the data storage and analysis module, the intelligent body module, the automatic decision module and the resource management module are electrically connected in sequence, and the resource management module is electrically connected to the intelligent body module; The communication protocol module is used to define and manage the communication standards and rules between agents according to the network environment and performance requirements, including identity authentication, synchronization mechanism and data encryption; The state perception module is used to collect network traffic data of network devices in real time based on sensors. The data includes bandwidth utilization, delay, packet loss rate, device temperature and power consumption, and detect network anomalies and provide early warning information. A data storage and analysis module is used to clean, preprocess and store the network flow data in the state perception module to obtain the preprocessed network flow data; The intelligent agent module is used to obtain time characteristics and optimal traffic adjustment strategies based on preprocessed network traffic data using deep neural networks and reinforcement learning models, and to update the strategies based on decision-making effect data; An automated decision-making module, which is used to generate network management decisions based on the results of the intelligent agent module, and automatically adjust network configuration, optimize resource allocation, and prevent potential failures according to the decisions; The resource management module is used to dynamically adjust the configuration of network resources, including bandwidth allocation, route selection and device scheduling, based on the results of the automated decision-making module and the intelligent agent module.

2. The communication network intelligent management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: The communication protocol module is configured to perform the following actions: Step 1: Use security protocol technology to authenticate the certificate or key provided by the agent, and exchange information after successful authentication; Step 2: Through information interaction, establish a synchronization mechanism to obtain and parse the data packets between agents. The data packets include timestamps and version information. The timestamp is used to identify the time point when the data packet is sent, and the version information is used to indicate the update status and version number of the data. The data information is sorted and integrated, the timing and status between agents are synchronized, and the data packets are encrypted; Step 3: Use cyclic redundancy check technology to check the integrity of the data packet. If the check result feedback is normal, no operation is performed. Otherwise, a data packet retransmission instruction is sent until the check result feedback is normal.

3. The communication network intelligent management system based on multi-agent reinforcement learning according to claim 2 is characterized in that: In step 3, verify Includes the following: Initialize the cyclic redundancy check register to 0, perform an XOR operation on each bit of the data and the current value of the cyclic redundancy check register, and shift the result left by one bit. If the highest bit after the left shift is 1, perform an XOR operation on the value and the polynomial to obtain the final cyclic redundancy check code, and append the final cyclic redundancy check code to the end of the original data and send it to the receiving end; calculate the cyclic redundancy check value of the original data and the cyclic redundancy check value of the data received by the receiving end. If the two are the same, the check passes, otherwise the check fails; The calculation formula of CRC value is: Among them, M(x) represents the original information polynomial, x represents the formal variable, n represents the total number of bits of the cyclic redundancy check code, k represents the number of bits of the original data, G(x) represents the generating polynomial, Q(x) represents the quotient of two polynomials, and R(x) represents the remainder polynomial.

4. The communication network intelligent management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: The data storage and analysis module is configured to perform the following actions: The network traffic data in the state perception module is cleaned to remove noise and outliers, and the cleaned data is converted into time series data; the cleaned data is compressed, normalized and feature extracted to obtain the preprocessed network traffic data.

5. The communication network intelligent management system based on multi-agent reinforcement learning according to claim 1 is characterized in that: The agent module is configured to perform the following actions: Step 1: Build a traffic pattern analysis model based on a deep neural network, input the preprocessed data into the model, and extract time features; Use historical network traffic data to train the traffic pattern analysis model, and use mean square error as the loss function to measure the prediction error; Input the preprocessed data into the trained traffic pattern analysis model to obtain the predicted traffic peak; Step 2: Based on real-time status and historical experience, use the reinforcement learning model to select the optimal action and adjust the traffic allocation rules; Step 3: Collect decision effect data through an event-based trigger mechanism and update the strategy based on the feedback results; events include time periodic events, traffic threshold events, performance indicator abnormal events and external trigger events; decision effect data includes network traffic conditions before and after decision execution, network performance indicators and user perception and feedback of network performance changes; The effectiveness of the decision is evaluated by comparing performance indicators before and after decision execution and analyzing changes in traffic patterns.

6. The communication network intelligent management system based on multi-agent reinforcement learning according to claim 5 is characterized in that: In step 2, adjusting the traffic distribution rules includes the following sub-steps: Step 201, abstracting the communication network into a reinforcement learning environment, where the real-time state consists of the network state and time characteristics, and the reward function is a measure of optimizing network performance; Step 202: Initialize network parameters and set two identical network structures, one for online prediction and the other for stabilizing the target value; input the network state into the reinforcement learning model and output the expected Q value of each action; Execute traffic adjustment strategies, observe state transitions and rewards, and store them in the experience replay pool; A batch of experiences is randomly drawn from the experience replay pool to train the reinforcement learning model; for each batch of experiences, the trained reinforcement learning model is used to predict the Q value of the current state, and the target network is used to predict the Q value of the next state to calculate the target Q value; The parameters of the trained reinforcement learning model are updated by minimizing the error between the predicted Q value and the target Q value, and the parameters of the trained reinforcement learning model are periodically copied to the target network; according to the current network state and the time features extracted in step 1, the Q value of each action is predicted by the trained reinforcement learning model, and the action with the highest Q value is selected as the optimal traffic adjustment strategy.

7. The communication network intelligent management system based on multi-agent reinforcement learning according to claim 5 is characterized in that: In step 3, the calculation formula for decision effect evaluation is: The formula for calculating the performance difference is: text difference value = index value after decision-index value before decision The formula for calculating the performance improvement percentage is: The formula for calculating the performance degradation percentage is: The formula for calculating the user satisfaction score is: Among them, A represents the total number of users participating in the feedback.

Images