Large model reinforcement learning network configuration generation method based on verification feedback

Abstract

The invention discloses a verification feedback-based large model reinforcement learning network configuration generation method, which comprises the following steps of: converting input network phenomena and state contents into a semantic and action sequence through network configuration semantic analysis; the method comprises the following steps: generating a configuration framework and parameters meeting requirements through a hybrid action space strategy generation and neural symbol collaborative reinforcement learning model, verifying and feeding back the generated network configuration information in a digital twin system, simulating a real network scene through formalized verification and performance simulation in a virtual environment, and establishing a network configuration model. According to the method, the correctness and the performance of configuration are comprehensively evaluated, then a feedback signal containing multi-dimensional information is generated, a strategy model is corrected and optimized according to digital twinborn verification, and finally, an intelligent agent is guided to adjust a high-entropy configuration item through a reward mechanism, and configuration is optimized to generate a large model.

Description

Technical Field

[0001] The present invention belongs to the intersection of artificial intelligence and network communication technology, and in particular relates to a large-model reinforcement learning network configuration generation method based on verification feedback. Background Art

[0002] With the rapid development of digital technology, communication networks are playing an increasingly important role in social life and industrial operations. From internal enterprise office networks to large data center networks supporting online services, to the mobile networks we use daily, their scale and complexity are growing exponentially. As network traffic increases, network configuration management faces enormous challenges. Traditional network configuration management relies primarily on manual operations, which is not only inefficient and prone to human error, but also inadequate for large-scale, dynamically changing network environments. Even small networks involve multiple devices and complex network policies, making the configuration process cumbersome and time-consuming. For large data centers and carrier networks, the difficulty of configuration management increases exponentially. Furthermore, with the widespread application of artificial intelligence, 5G, the Internet of Things, and cloud computing, the demand for network dynamism and flexibility has increased significantly. Networks must rapidly adjust and reconfigure based on real-time service demands and changes in user behavior.

[0003] Existing automated communication network configuration technologies rely on predefined policy templates. Fixed templates are difficult to adapt to sudden demand. In the event of traffic bursts or sudden topology changes, policy implementation delays exceed the service tolerance threshold (measured as 12-15 seconds), leading to a surge in the risk of QoS degradation. Open-source verification tools (such as Batfish) can detect configuration conflicts, but they use an offline analysis model and cannot be integrated into the configuration generation loop. Some current research uses deep learning models (such as those based on GNNs) for communication network configuration generation. However, model training relies on historical static datasets, which is temporally mismatched with dynamic network environments. This results in policy generation accuracy decreasing as network state changes. Therefore, a large-scale reinforcement learning network configuration generation method based on verification feedback is needed to address this issue. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a large-model reinforcement learning network configuration generation method based on verification feedback, which aims to solve the problems of the existing network communication automation configuration relying on predefined templates, being relatively rigid, unable to cope with sudden demands outside the templates, having high policy generation delay, easily causing system risks and poor accuracy; it has the characteristics of being able to obtain network status feedback through real-time environment interaction, breaking through the response speed limitation of the static rule base, and enabling the system to continuously adapt to the evolution of network topology and the migration of business models.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for generating a large-model reinforcement learning network configuration based on verification feedback includes the following steps: S1, semantic parsing based on the network configuration of graph neural network: Convert the input unstructured business requirements into semantic action sequences, build a formal ontology library and dynamic knowledge graph, and combine it with a time-series graph convolutional network to update the network status in real time; S2, a hybrid action space construction strategy generation model for communication network configuration: Adopting a hierarchical neural symbolic reinforcement learning method, symbolic rules and continuous parameters are collaboratively optimized to generate a configuration framework that includes device parameters and link connections. S3, strategy optimization based on communication network configuration calculated by causal graph: The generated configuration is input into the digital twin system for simulation verification, the node causal entropy is calculated based on the causal graph, high entropy nodes are selected, and the strategy is optimized through reinforcement learning; S4, build strategy to generate model optimization: Based on the causal entropy value and feedback signal, the high entropy node configuration is adjusted, and the configuration strategy is adjusted through the reinforcement learning algorithm to optimize the overall operation status of the network, and finally the optimized configuration is generated into a large model.

[0006] Preferably, in step S1, semantic analysis specifically includes: Adopt OWL-based method to build formal ontology library, define equipment, protocol and topological entity relationship; Use the time-series graph convolutional network to update link load and device performance status in real time: ; Where, Indicates that at time step t The hidden state matrix of the nodes in the graph is, is the activation function, is the normalized adjacency matrix, Contains node-level features. is the learnable graph / node feature transformation matrix, b is the bias vector.

[0007] Preferably, in step S2, strategy generation specifically includes: The probability of symbolic rule selection is calculated using the probabilistic logic programming formula: ; Where, Indicates that the status , select Symbol Rules The probability of adopting a specific routing strategy or other symbolic rules under a specific network state; Represents the current operating status of the network. Indicates a specific network configuration rule. represents the total number of rules in the symbolic rule set; Is a function that measures the state Next, the symbol rule The higher the score, the more likely the rule is to be selected in the current state.

[0008] Preferably, the strategy generation further includes a dynamic weight fusion module, which automatically adjusts the contribution ratio of the symbol rule and the continuous parameter optimization according to the degree of network load fluctuation.

[0009] Preferably, in step S3, the strategy optimization specifically includes: Each configuration item in the network, including bandwidth allocation and routing settings, is used as a node , the mutual influence relationship between configuration items is used as the edge , build a configuration item cause-and-effect diagram Perform causal analysis and extract configuration item dependencies from verification logs; For each configuration item node , determine the set of all possible states or events related to , and estimate each state or event The probability of occurrence; the causal entropy of the node is calculated by the following formula: ; Where, Representation node The causal entropy of is a random variable representing the relationship between the node A state or event related to Is with the node The set of all possible states or events related to it, Indicates that the node Related context, state or event Probability of occurrence.

[0010] Preferably, in step S4, model optimization specifically includes: The configuration strategy is adjusted through reinforcement learning algorithms to optimize the overall operation status of the network, and ultimately the configuration is optimized to generate a large model.

[0011] Preferably, the verification process of the digital twin system includes: Formal verification of configuration conflict detection and performance simulation simulating real network scenarios; Generate multi-dimensional feedback signals including latency, throughput, and packet loss rate for policy optimization.

[0012] Preferably, the construction of the causal diagram further includes: Extract configuration item dependencies from verification logs and dynamically update edge EE weights based on historical data and real-time status; Adapt to network topology evolution and business model migration through incremental model update algorithm.

[0013] Preferably, the reward mechanism for reinforcement learning includes: Set the reward function based on the node causal entropy value, and prioritize the configuration adjustment actions of high entropy nodes; The reward value is positively correlated with network performance indicators including QoS and load balancing, and negatively correlated with the number of configuration conflicts.

[0014] Preferably, natural language processing (NLP) is introduced in the semantic parsing stage to parse the fuzzy semantics in unstructured requirements; adversarial training is used to enhance the robustness of graph neural networks and avoid parsing deviations caused by noisy data.

[0015] The beneficial effects of the present invention are as follows: 1. This method first accurately converts the input network phenomena and state content into semantics and action sequences that can be understood by the intelligent agent in reinforcement learning through network configuration semantic analysis. Then, based on these semantic action sequences, through hybrid action space strategy generation, the neural symbolic collaborative reinforcement learning model is used to fully consider the network constraints and generate a configuration framework and parameters that meet the requirements, providing a specific implementation plan for network configuration. Subsequently, the generated network configuration information is verified and feedbacked in the digital twin system. Through formal verification and performance simulation in a virtual environment, the real network scenario is simulated to comprehensively evaluate the correctness and performance of the configuration, and then generate a feedback signal containing multi-dimensional information. Finally, in the configuration strategy optimization stage, the strategy model is corrected and optimized based on the feedback obtained from the digital twin verification, and the configuration strategy is adjusted through the reinforcement learning algorithm to optimize the overall operation status of the network, and finally the configuration is optimized to generate a large model.

[0016] 2. This method has the capabilities of "environmental perception-intelligent decision-making-closed-loop verification". It applies a dynamic policy generation mechanism driven by reinforcement learning, obtains network status feedback through real-time environmental interaction, and breaks through the response speed limitation of the static rule base. At the same time, it develops an incremental model update algorithm to enable the system to continuously adapt to the evolution of network topology and the migration of business models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of a flow chart of the present invention; Figure 2 It is a schematic diagram of the entire process in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Example 1: like Figure 1 As shown, a method for generating a large-model reinforcement learning network configuration based on verification feedback includes the following steps: S1, semantic parsing based on the network configuration of graph neural network: Convert the input unstructured business requirements into semantic action sequences, build a formal ontology library and dynamic knowledge graph, and combine it with a time-series graph convolutional network to update the network status in real time; S2, a hybrid action space construction strategy generation model for communication network configuration: Adopting a hierarchical neural symbolic reinforcement learning method, symbolic rules and continuous parameters are collaboratively optimized to generate a configuration framework that includes device parameters and link connections. S3, strategy optimization based on communication network configuration calculated by causal graph: The generated configuration is input into the digital twin system for simulation verification, the node causal entropy is calculated based on the causal graph, high entropy nodes are selected, and the strategy is optimized through reinforcement learning; S4, build strategy to generate model optimization: Based on the causal entropy value and feedback signal, the high entropy node configuration is adjusted, and the configuration strategy is adjusted through the reinforcement learning algorithm to optimize the overall operation status of the network, and finally the optimized configuration is generated into a large model.

[0019] Example 2: like Figure 2 As shown, this embodiment proposes a digital twin verification feedback reinforcement learning optimization framework that integrates semantic parsing, hybrid action space strategy generation, digital twin verification and feedback-driven, and realizes autonomous generation and dynamic optimization of network configuration through innovative technologies such as neural symbolic collaboration, causal reasoning and quantum acceleration.

[0020] Preferably, semantic analysis specifically includes: Adopt OWL-based method to build formal ontology library, define equipment, protocol and topological entity relationship; Use the time-series graph convolutional network to update link load and device performance status in real time: ; Where, Indicates that at time step t The hidden state matrix of the nodes in the graph is, is the activation function, is the normalized adjacency matrix, Contains node-level features. is the learnable graph / node feature transformation matrix, b is the bias vector.

[0021] Preferably, strategy generation specifically includes: The probability of symbolic rule selection is calculated using the probabilistic logic programming formula: ; Where, Indicates that the status , select Symbol Rules The probability of adopting a specific routing strategy or other symbolic rules under a specific network state; Represents the current operating status of the network. Indicates a specific network configuration rule. represents the total number of rules in the symbolic rule set; Is a function that measures the state Next, the symbol rule The higher the score, the more likely the rule is to be selected in the current state.

[0022] Preferably, the strategy generation further includes a dynamic weight fusion module, which automatically adjusts the contribution ratio of the sign rule and the continuous parameter optimization according to the degree of network load fluctuation.

[0023] Preferably, the strategy optimization specifically includes: Each configuration item in the network, including bandwidth allocation and routing settings, is used as a node , the mutual influence relationship between configuration items is used as the edge , build a configuration item cause-and-effect diagram Perform causal analysis and extract configuration item dependencies from verification logs; For each configuration item node , determine the set of all possible states or events related to , and estimate each state or event The probability of occurrence; the causal entropy of the node is calculated by the following formula: ; Where, Representation node The causal entropy of is a random variable representing the relationship between the node A state or event related to Is with the node The set of all possible states or events related to Indicates that the node Related context, state or event Probability of occurrence.

[0024] Preferably, model optimization specifically includes: Adjust the configuration strategy through reinforcement learning algorithm to optimize the overall operation status of the network, and finally optimize the configuration to generate a large model; Preferably, the verification process of the digital twin system includes: Formal verification of configuration conflict detection and performance simulation simulating real network scenarios; Generate multi-dimensional feedback signals including latency, throughput, and packet loss rate for policy optimization.

[0025] Preferably, the construction of the causal diagram further includes: Extract configuration item dependencies from verification logs and dynamically update edge EE weights based on historical data and real-time status; Adapt to network topology evolution and business model migration through incremental model update algorithm.

[0026] Preferably, the reward mechanism for reinforcement learning includes: Set the reward function based on the node causal entropy value, and prioritize the configuration adjustment actions of high entropy nodes; The reward value is positively correlated with network performance indicators including QoS and load balancing, and negatively correlated with the number of configuration conflicts.

[0027] Preferably, natural language processing (NLP) is introduced in the semantic parsing stage to parse the fuzzy semantics in unstructured requirements; adversarial training is used to enhance the robustness of graph neural networks and avoid parsing deviations caused by noisy data.

[0028] Example 3: Data Center Network Burst Traffic Scheduling Scenario Example Background: In a data center with a 200-server, three-tier Fat-Tree topology, the access layer directly connects to the 200 servers. Under normal circumstances, network traffic between servers remains relatively stable, with each server requiring approximately 200 Mbps of bandwidth. However, in emergencies, such as during an urgent video conference, access layer network traffic can surge dramatically. Normal inter-server traffic is disrupted, and the bandwidth requirement for each server suddenly increases from the baseline 200 Mbps to 800 Mbps. The access layer switches face a massive traffic surge and must fully handle this explosive growth. The core layer has a multi-path routing mechanism. When bursts of access layer traffic converge on the core layer, the core switches must make swift and accurate decisions. Based on multiple factors, such as real-time network status and link load, the core switches must select the optimal transmission path for the massive data flow. This ensures fast and stable data transmission, avoids network congestion and data loss, and ensures the smooth progress of the emergency video conference. Implementation process: (1) Semantic analysis First, semantic parsing of the network status is performed. The input is an unstructured requirement: "Prioritize video conferencing bandwidth and automatically avoid heavily loaded links." This requirement is relatively abstract, requiring a graph neural network-based semantic parsing method to analyze the input text, understand the user's intent at a semantic level, and generate a specific semantic action sequence: [Bandwidth allocation priority: video conferencing > 80%; Link load threshold: 75%; Routing strategy: ECMP optimization]. "Bandwidth allocation priority: video conferencing > 80%" indicates that when allocating network bandwidth, video conferencing bandwidth must exceed 80% of the total bandwidth to prioritize video conferencing. "Link load threshold: 75%" sets the link load threshold. When the link load reaches or exceeds 75%, it is considered a heavily loaded link and the system will automatically avoid it. "Routing strategy: ECMP optimization" specifies that the equal-cost multi-path (ECMP) optimization strategy will be used in routing to ensure optimal traffic distribution.

[0029] During this process, a temporal graph convolutional network (GCNN) was used to predict link states. The network parameters were set to 64 node feature dimensions and an adjacency matrix update frequency of 1 Hz. By learning and training on historical link state data, the final prediction results provided a reliable basis for subsequent strategy generation and network optimization.

[0030] (2) Strategy Generation After completing the semantic analysis, we enter the strategy generation phase, which mainly includes three parts: symbol rule selection, continuous parameter optimization, and generation of specific configurations.

[0031] Symbolic Rule Selection: The system's rule base contains 32 predefined routing policies, designed for different network scenarios and requirements. Under the current load surge, extensive experiments and data analysis have shown that ECMP optimization rules have a higher probability of success than the fixed selections in traditional static rule bases. ECMP optimization rules can more effectively respond to network changes and achieve better traffic distribution and load balancing, and are therefore the preferred routing policy.

[0032] Continuous Parameter Optimization: To further optimize network performance, bandwidth allocation weighting factors are dynamically adjusted. During operation, bandwidth allocation weighting factors are adjusted incrementally based on changes in network traffic, with a response time of less than 50ms. This rapid dynamic adjustment adapts to fluctuations in network traffic, ensuring optimal bandwidth allocation under varying network loads and safeguarding the normal operation of critical services such as video conferencing.

[0033] Generate a specific configuration: Based on the aforementioned symbol rule selection and continuous parameter optimization, the final network configuration is generated: {Guaranteed video conferencing bandwidth: 650 Mbps, Redundant paths: 3, QoS level: Gold}. "Guaranteed video conferencing bandwidth: 650 Mbps" specifies the minimum guaranteed bandwidth for video conferencing to ensure smooth video conferencing; "Redundant paths: 3" sets three redundant paths. If the primary path fails or is congested, data can be transmitted over these redundant paths, improving network reliability; and "QoS level: Gold" assigns a higher quality of service (QoS) to video conferencing services, ensuring their priority transmission within the network.

[0034] (3) Strategy optimization After the policy is generated, it needs to be optimized to ensure its optimal performance in the actual network environment. Policy optimization is mainly achieved through two aspects: digital twin verification indicators and causal entropy analysis.

[0035] Digital Twin Verification Metrics: A digital twin system was used to compare and analyze key network performance indicators before and after policy optimization. In terms of latency, the network latency before optimization was significantly greater than after policy optimization. This optimization resulted in faster data transmission within the network, providing a smoother user experience. In terms of throughput, the optimization significantly improved the network's data transmission capacity, better meeting user demands for high-bandwidth services. In terms of packet loss rate, the optimization reduced data loss and improved data transmission reliability.

[0036] Causal entropy analysis: Causal entropy analysis is performed on core switches. The causal entropy value reflects the complexity and uncertainty of the switch in the network. When the value is too high, it can cause problems such as network jitter. By optimizing the ECMP weight distribution, network jitter is effectively reduced. This shows that causal entropy analysis and targeted optimization measures can further improve network stability and performance.

[0037] (4) Model optimization First, high-entropy nodes are screened, and the causal entropy threshold is determined using Monte Carlo simulation. Monte Carlo simulation, a statistical method based on random sampling, simulates various network states through a large number of random experiments to determine an appropriate threshold. Then, the causal entropy is calculated for all nodes in the network, and nodes with causal entropy greater than the threshold are screened. To guide the agent to prioritize the optimization of high-entropy nodes, a dynamic reward function is designed to focus on improving network service quality; a penalty coefficient is used to penalize configuration conflicts. A 3x reward weight is applied to high-entropy nodes, making the agent more inclined to operate on these nodes during the optimization process. When the agent's actions reduce the causal entropy of high-entropy nodes, improve network QoS, and reduce configuration conflicts, it will receive a higher reward, thus incentivizing the agent to continuously explore the optimal optimization strategy and achieve model optimization.

[0038] After a comprehensive optimization of the network system, core performance indicators have significantly improved. Network latency has been reduced from over 50ms to less than 30ms, throughput has increased to 8.2Tbps, a 37% increase, and packet loss has been effectively controlled from 1.2% to less than 0.15%. Key business support has been strengthened, with video conferencing bandwidth consistently maintained at over 650Mbps. Three redundant paths have been established, achieving 99.99% transmission reliability. Quality of service meets Gold-level standards, with latency jitter less than 5ms.

Claims

1. A method for generating a large-scale reinforcement learning network configuration based on verification feedback, characterized in that: The following steps are involved: S1, semantic parsing based on the network configuration of graph neural network: Convert the input unstructured business requirements into semantic action sequences, build a formal ontology library and dynamic knowledge graph, and combine it with a time-series graph convolutional network to update the network status in real time; S2, a hybrid action space construction strategy generation model for communication network configuration: Adopting a hierarchical neural symbolic reinforcement learning method, symbolic rules and continuous parameters are collaboratively optimized to generate a configuration framework that includes device parameters and link connections. S3, strategy optimization based on communication network configuration calculated by causal graph: The generated configuration is input into the digital twin system for simulation verification, the node causal entropy is calculated based on the causal graph, high entropy nodes are selected, and the strategy is optimized through reinforcement learning; S4, build strategy to generate model optimization: Based on the causal entropy value and feedback signal, the high entropy node configuration is adjusted, and the configuration strategy is adjusted through the reinforcement learning algorithm to optimize the overall operation status of the network, and finally the optimized configuration is generated into a large model.

2. A method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: In step S1, semantic analysis specifically includes: Adopt OWL-based method to build formal ontology library, define equipment, protocol and topological entity relationship; Use the time-series graph convolutional network to update link load and device performance status in real time: ； Where, Indicates that at time step t The hidden state matrix of the nodes in the graph is, is the activation function, is the normalized adjacency matrix, Contains node-level features. is the learnable graph / node feature transformation matrix, b is the bias vector.

3. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: In step S2, strategy generation specifically includes: The probability of symbolic rule selection is calculated using the probabilistic logic programming formula: ； Where, Indicates that the status , select Symbol Rules The probability of adopting a specific routing strategy or other symbolic rules under a specific network state; Represents the current operating status of the network. Indicates a specific network configuration rule. represents the total number of rules in the symbolic rule set; Is a function that measures the state Next, the symbol rule The higher the score, the more likely the rule is to be selected in the current state.

4. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 3, characterized in that: Strategy generation also includes a dynamic weight fusion module that automatically adjusts the contribution ratio of symbolic rules and continuous parameter optimization according to the degree of network load fluctuation.

5. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: In step S3, the strategy optimization specifically includes: Each configuration item in the network, including bandwidth allocation and routing settings, is used as a node , the mutual influence relationship between configuration items is used as the edge , build a configuration item cause-and-effect diagram Perform causal analysis and extract configuration item dependencies from verification logs; For each configuration item node , determine the set of all possible states or events related to , and estimate each state or event The probability of occurrence; the causal entropy of the node is calculated by the following formula: ； Where, Representation node The causal entropy of is a random variable representing the relationship between the node A state or event related to Is with the node The set of all possible states or events related to Indicates that the node Related context, state or event Probability of occurrence.

6. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: In step S4, model optimization specifically includes: The configuration strategy is adjusted through reinforcement learning algorithm to optimize the overall operation status of the network, and finally the configuration is optimized to generate a large model.

7. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: The verification process of the digital twin system includes: Formal verification of configuration conflict detection and performance simulation simulating real network scenarios; Generate multi-dimensional feedback signals including latency, throughput, and packet loss rate for policy optimization.

8. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 5, characterized in that: The construction of the causal diagram also includes: Extract configuration item dependencies from verification logs and dynamically update edge EE weights based on historical data and real-time status; Adapt to network topology evolution and business model migration through incremental model update algorithm.

9. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: The reward mechanism of reinforcement learning includes: Set the reward function based on the node causal entropy value, and prioritize the configuration adjustment actions of high entropy nodes; The reward value is positively correlated with network performance indicators including QoS and load balancing, and negatively correlated with the number of configuration conflicts.

10. The method for generating a large-scale reinforcement learning network configuration based on verification feedback according to claim 1, characterized in that: Natural language processing (NLP) is introduced in the semantic parsing stage to parse the fuzzy semantics in unstructured requirements; adversarial training is used to enhance the robustness of graph neural networks and avoid parsing biases caused by noisy data.

Images